{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JS6jP7tpCrLr"
   },
   "source": [
    "## Digit Recognizer\n",
    "Learn computer vision fundamentals with the famous MNIST dat\n",
    "\n",
    "https://www.kaggle.com/c/digit-recognizer\n",
    "\n",
    "### Competition Description\n",
    "MNIST (\"Modified National Institute of Standards and Technology\") is the de facto “hello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.\n",
    "\n",
    "In this competition, your goal is to correctly identify digits from a dataset of tens of thousands of handwritten images. We’ve curated a set of tutorial-style kernels which cover everything from regression to neural networks. We encourage you to experiment with different algorithms to learn first-hand what works well and how techniques compare.\n",
    "\n",
    "### Practice Skills\n",
    "Computer vision fundamentals including simple neural networks\n",
    "\n",
    "Classification methods such as SVM and K-nearest neighbors\n",
    "\n",
    "#### Acknowledgements \n",
    "More details about the dataset, including algorithms that have been tried on it and their levels of success, can be found at http://yann.lecun.com/exdb/mnist/index.html. The dataset is made available under a Creative Commons Attribution-Share Alike 3.0 license."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LgcAAVmXgBPm"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import math\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt, matplotlib.image as mpimg\n",
    "from sklearn.model_selection import train_test_split\n",
    "import tensorflow as tf\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EITzxKZRgBPy"
   },
   "outputs": [],
   "source": [
    "from tensorflow import keras\n",
    "from tensorflow.keras import models\n",
    "from tensorflow.keras import losses,optimizers,metrics\n",
    "from tensorflow.keras import layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4PihLjAggBP1"
   },
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "cra90fdEsmsI",
    "outputId": "1b5e320d-9f3f-4571-a3b3-88564bcd7270"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Drive already mounted at /content/gdrive; to attempt to forcibly remount, call drive.mount(\"/content/gdrive\", force_remount=True).\n"
     ]
    }
   ],
   "source": [
    "from google.colab import drive \n",
    "drive.mount('/content/gdrive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "K58DeQH2gBP2"
   },
   "outputs": [],
   "source": [
    "labeled_images = pd.read_csv('gdrive/My Drive/dataML/train.csv')\n",
    "#labeled_images = pd.read_csv('train.csv')\n",
    "images = labeled_images.iloc[:,1:]\n",
    "labels = labeled_images.iloc[:,:1]\n",
    "train_images, test_images,train_labels, test_labels = train_test_split(images, labels, test_size=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "id": "At9soAW0qOs4",
    "outputId": "949d98c0-2251-4adc-e1a1-c7aa980cca2e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(41580, 784)\n",
      "(41580, 1)\n",
      "(420, 784)\n",
      "(420, 1)\n"
     ]
    }
   ],
   "source": [
    "print(train_images.shape)\n",
    "print(train_labels.shape)\n",
    "print(test_images.shape)\n",
    "print(test_labels.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "f87cEn1_xfqI"
   },
   "source": [
    "## Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UfkBgDM1gBP5"
   },
   "source": [
    "#### convert the data to the right type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-X3Uu-o_gBP6"
   },
   "outputs": [],
   "source": [
    "x_train = train_images.values.reshape(train_images.shape[0],28,28,1)/255\n",
    "x_test = test_images.values.reshape(test_images.shape[0],28,28,1)/255\n",
    "y_train = train_labels.values\n",
    "y_test = test_labels.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 282
    },
    "colab_type": "code",
    "id": "7vboLIlsgBP9",
    "outputId": "a2428797-704d-4688-ffe1-74f9f691c258"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f1842b17c50>"
      ]
     },
     "execution_count": 59,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPoAAAD4CAYAAADFJPs2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADhJJREFUeJzt3VuMXdV9x/GvcY2CrJLmotoJisFD\no79q8RQejBFOnIbUYLXlAUfB4iZAclWbKFIVS44QEiBoTJBFVbBBUdpQuQrCCCkYx1xstwoPRgpC\nTZSgaBFsiwcbYyBKDE3lDmP6MMejM8dz9h6fu+f//bxw9l6zzvx9Zn7sy9pr1ryPP/4YSXPbecMu\nQFL/GXQpAYMuJWDQpQQMupTAnwzo+3hrX+q/ee0aOg56RDwMXMFkiL9dSnm10/eS1F8dnbpHxFeA\nL5ZSVgB3AP/S06ok9VSn1+hfA34CUEr5DfCpiLiwZ1VJ6qlOg74YeLdp+93GPkkjqFd33dveBJA0\nfJ0G/SjTj+CfB97uvhxJ/dBp0F8C1gJExJeAo6WUD3pWlaSemtfp7LWI2AJ8GTgFbCyl/LLiyx1H\nl/qv7SV0x0E/SwZd6r+2QfcRWCkBgy4lYNClBAy6lIBBlxIw6FICBl1KwKBLCRh0KQGDLiVg0KUE\nDLqUgEGXEjDoUgIGXUrAoEsJGHQpAYMuJWDQpQQMupSAQZcSGNSyyRoxhw4dqmw/duxYZfv9998/\nbXvPnj2sWbNmavv5559v2/e886qPL6dOnapsf/DBByvb165dO217bGxs6t87NjZW2Xeu8oguJWDQ\npQQMupSAQZcSMOhSAgZdSsCgSwm4mmofbdiwobJ93ry2i18C0Pqz2b59+7T3rOpf93N96aWXKtsP\nHz5c2d46Fj4+Ps6CBQumtqvGwrsdR6/rv379+mnb27ZtY+PGjVOv57C2vxAdPTATEauAp4HXG7t+\nVUr5VifvJan/unky7mellLX1XyZp2LxGlxLo6Bq9ceq+HXgT+DRwbyllb0WXlNfo0oC1vUbvNOgX\nAVcBO4Ex4L+Avyil/F+bLimD7s242fVt5c24jvX2Zlwp5QjwVGPzYEQcAy4Cqn87JA1FR9foEXFj\nRHyn8XoxsAg40svCJPVOp3fddwE/jojrgPOBf6g4bR+qEydOVLbfcMMN07YHOa/6bPtv376dxx57\nbFb9607dz/ayoa621n1V/es+l06+d13/AT0vMrI6PXX/APjbHtciqU8cXpMSMOhSAgZdSsCgSwkY\ndCmBOf/nnl9//fXK9r17z3xyt3lf1RBW3fBYnU76N/ep6r969erK97n77rsr2zsZnjtw4MCs+rf+\nqehWVUOaUP+5vfjii2331T3xt3Tp0sr2c5VHdCkBgy4lYNClBAy6lIBBlxIw6FICBl1KYM6Poy9a\ntKiyfcmSJZX7qpYXvvbaa8/6vZtt2rSpsn2mMd2JiYnKPsO0fPnyWX1d3efS7TTVmZ4hOL1vro6T\n1/GILiVg0KUEDLqUgEGXEjDoUgIGXUrAoEsJzPlx9LGxscr2ffv2Ve5755132vZdtmxZ5XtfeOGF\nNdXlVPenpuva6+ajz9S/7j3nOo/oUgIGXUrAoEsJGHQpAYMuJWDQpQQMupTAvAEtJ5t7zdqEtmzZ\n0rbtrrvuquxb9zvZvKz1THbv3l3ZPoe1fVhgVg/MRMRlwLPAw6WURyPiC8AOYD7wNnBzKeVkLyqV\n1Hu1p+4RsRB4BNjftPs+YFspZSXwJnB7f8qT1AuzuUY/CawBjjbtWwXsarx+Dri6t2VJ6qXaU/dS\nykfARxHRvHth06n6ceBzfahN57DNmzd31Kb+6MWkltyzBTQjb8aNlk6H1z6MiAsary9i+mm9pBHT\nadD3Adc3Xl8PvNCbciT1Q+04ekRcDmwFLgHGgSPAjcATwCeAt4DbSinjFW/jOPoc8+STT07bXrdu\n3bR9N910U9u+dfPJ6/5ue/M67DOZ7d+Xn4M6H0cvpbzG5F32Vl/voiBJA+QjsFICBl1KwKBLCRh0\nKQGDLiXgNFXNqOrJNjjz6baJiQnmz58/tV31e1X3p5cvvvjiyvb9+/dXtmddGpmK4TWP6FICBl1K\nwKBLCRh0KQGDLiVg0KUEDLqUgOPoSbVOM21VNc0UzpxqOj4+zoIFC6a2q6aa1k1TfeONNyrbE4+T\n13EcXcrMoEsJGHQpAYMuJWDQpQQMupSAQZcS6MVKLRpBJ06cqGzfsWNHZXvd8xUzjZM376uaU+58\n8sHziC4lYNClBAy6lIBBlxIw6FICBl1KwKBLCTgf/RxWNae8bpx87969le11SxfXzUevmlPuOHnf\ndL5sMkBEXAY8CzxcSnk0Ip4ALgfeb3zJQ6WUn3ZbpaT+qA16RCwEHgFaH2f6billd1+qktRTs7lG\nPwmsAY72uRZJfTLra/SIuAd4r+nUfTFwPnAcuLOU8l5Fd6/Rpf7r7hp9BjuA90spv4iIzcA9wJ0d\nvpc65M04zVZHQS+lNF+v7wIe6005kvqho3H0iHgmIsYam6uAX/esIkk9V3uNHhGXA1uBS4Bx4AiT\nd+E3A38EPgRuK6Ucr3gbr9E70LpG+ebNm6fta12jvFndz7XXa5QvXbqUw4cPT9vWwHV+jV5KeY3J\no3arZ7ooSNIA+QislIBBlxIw6FICBl1KwKBLCThNdYjOduniiYkJ5s+fP7Vdtfzw2T7Z1sqli89J\nLpssZWbQpQQMupSAQZcSMOhSAgZdSsCgSwm4bHIfvfLKK5XtnSxd3Lyvaqy87vmIBx54oLL92LFj\nZ9W+YsWK2n/voCxatGja9tjYGIcOHZp6nZFHdCkBgy4lYNClBAy6lIBBlxIw6FICBl1KwPnoXTjb\n+eSt6uaEt46TD3I+ercrtVT17/X3brVkyZJp2wcPHuTSSy8F4Jprrqnse9VVV1W2r1u3rrJ9yJyP\nLmVm0KUEDLqUgEGXEjDoUgIGXUrAoEsJpB9HPz1P+bTmucsAO3fubNu3atli6H7p4tb+p06dmjaG\nXNW/19+7rn/rGH9V/15/77r+zZ9bt9/7wIEDle1XXHFFZXufdb5sMkBEfB9Y2fj67wGvAjuA+cDb\nwM2llJPd1ympH2pP3SPiq8BlpZQVwDXAPwP3AdtKKSuBN4Hb+1qlpK7M5hr9ZeAbjde/BxYCq4Bd\njX3PAVf3vDJJPXNW1+gRsZ7JU/jVpZQ/b+y7FNhRSrmyouvIXqNLc0h31+gAEXEdcAfw18BvZ/Pm\n5wJvxvWmvzfjJg35Zlxbsxpei4jVwF3AtaWUPwAfRsQFjeaLgKN9qk9SD9Qe0SPik8BDwNWllN81\ndu8Drgf+o/HfF/pWYZ9t3bp12va2bdum7Xv88cfb9u33dMuZ+jcfkbqZprply5bK9pUrV1a2z3Rk\nbD7aDfOIfuWVZ15Fnu7T7c+s7nuPqtmcun8T+CywMyJO77sV+GFE/D3wFvDv/SlPUi/UBr2U8gPg\nBzM0fb335UjqBx+BlRIw6FICBl1KwKBLCRh0KYH0yybXLU1cNaZbN+ZaNx68evXqyvbdu3efsW9i\nYqKyzzAtX7582CUAM39Gp/dt2LChsm/VcxNQ/zMdVR7RpQQMupSAQZcSMOhSAgZdSsCgSwkYdCmB\n9OPoM80vbt5XNf+42zndy5Ytq6lOvbZp06bK9ltuuaWy/Vz9mXlElxIw6FICBl1KwKBLCRh0KQGD\nLiVg0KUE0i+bLM0hbR/68IguJWDQpQQMupSAQZcSMOhSAgZdSsCgSwnMaj56RHwfWNn4+u8Bfwdc\nDrzf+JKHSik/7UuFkrpWG/SI+CpwWSllRUR8Bvhv4D+B75ZSzlxhQNLImc0R/WXg543XvwcWAvP7\nVpGknjurR2AjYj2Tp/ATwGLgfOA4cGcp5b2Krj4CK/Vf94/ARsR1wB3AncAOYHMp5a+AXwD3dFmg\npD6a7c241cBdwDWllD8A+5uadwGP9aE2ST1Se0SPiE8CDwF/U0r5XWPfMxEx1viSVcCv+1ahpK7N\n5oj+TeCzwM6IOL3vR8BTEfFH4EPgtv6UJ6kXnI8uzR3OR5cyM+hSAgZdSsCgSwkYdCkBgy4lYNCl\nBAy6lIBBlxIw6FICBl1KwKBLCRh0KQGDLiUwq78w0wNtp89J6j+P6FICBl1KwKBLCRh0KQGDLiVg\n0KUEDLqUwKDG0adExMPAFUz+Cehvl1JeHXQNM4mIVcDTwOuNXb8qpXxreBVBRFwGPAs8XEp5NCK+\nwORyWPOBt4GbSyknR6S2JxiRpbRnWOb7VUbgcxvm8uMDDXpEfAX4YmMJ5r8E/g1YMcgaavyslLJ2\n2EUARMRC4BGmL391H7CtlPJ0RPwTcDtDWA6rTW0wAktpt1nmez9D/tyGvfz4oE/dvwb8BKCU8hvg\nUxFx4YBrOFecBNYAR5v2rWJyrTuA54CrB1zTaTPVNipeBr7ReH16me9VDP9zm6mugS0/PuhT98XA\na03b7zb2nRhwHe0si4hdwKeBe0spe4dVSCnlI+CjpmWwABY2nXIeBz438MJoWxvAnRHxj8xuKe1+\n1TYB/E9j8w5gD7B62J9bm7omGNBnNuybcaP0DPxvgXuB64BbgX+NiPOHW1KlUfrsYMSW0m5Z5rvZ\nUD+3YS0/Pugj+lEmj+CnfZ7JmyNDV0o5AjzV2DwYEceAi4DDw6vqDB9GxAWllP9lsraROXUupYzM\nUtqty3xHxEh8bsNcfnzQR/SXgLUAEfEl4Ggp5YMB1zCjiLgxIr7TeL0YWAQcGW5VZ9gHXN94fT3w\nwhBrmWZUltKeaZlvRuBzG/by44NaTXVKRGwBvgycAjaWUn450ALaiIg/BX4M/BlwPpPX6HuGWM/l\nwFbgEmCcyf/p3Ag8AXwCeAu4rZQyPiK1PQJsBqaW0i6lHB9CbeuZPAV+o2n3rcAPGeLn1qauHzF5\nCt/3z2zgQZc0eMO+GSdpAAy6lIBBlxIw6FICBl1KwKBLCRh0KYH/BwwNtAs+FtNSAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "plt.imshow(x_train[12].squeeze())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "o5tcP2-KVHu-",
    "outputId": "4ea52c49-87c4-42f3-b7f4-0167b9e1eeb8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(41580, 28, 28, 1)"
      ]
     },
     "execution_count": 60,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "b6I1adl5gBQD"
   },
   "source": [
    "#### convert the data to the right type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "collapsed": true,
    "id": "xeMZR7ntgBQI"
   },
   "source": [
    "### convert class vectors to binary class matrices - this is for use in the\n",
    "### categorical_crossentropy loss below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8e2qjHPLgBQJ"
   },
   "outputs": [],
   "source": [
    "y_train = keras.utils.to_categorical(y_train)\n",
    "y_test = keras.utils.to_categorical(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LT78eGccgBQN"
   },
   "source": [
    "### Creating the Model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 374
    },
    "colab_type": "code",
    "id": "AMOStnPCFWSI",
    "outputId": "d2235c6f-f03c-4603-ce84-aaa7f06bc680"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_9 (Conv2D)            (None, 28, 28, 12)        444       \n",
      "_________________________________________________________________\n",
      "conv2d_10 (Conv2D)           (None, 14, 14, 24)        7224      \n",
      "_________________________________________________________________\n",
      "conv2d_11 (Conv2D)           (None, 7, 7, 48)          18480     \n",
      "_________________________________________________________________\n",
      "flatten_3 (Flatten)          (None, 2352)              0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 200)               470600    \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 200)               0         \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 10)                2010      \n",
      "=================================================================\n",
      "Total params: 498,758\n",
      "Trainable params: 498,758\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "\n",
    "model.add(layers.Conv2D(filters = 12, kernel_size=(6,6), strides=(1,1),\n",
    "                       padding = 'same', activation = 'relu',\n",
    "                       input_shape = (28,28,1)))\n",
    "          \n",
    "model.add(layers.Conv2D(filters = 24,kernel_size=(5,5),strides=(2,2),\n",
    "                       padding = 'same', activation = 'relu'))\n",
    "\n",
    "model.add(layers.Conv2D(filters = 48,kernel_size=(4,4),strides=(2,2),\n",
    "                       padding = 'same', activation = 'relu'))\n",
    "          \n",
    "model.add(layers.Flatten())          \n",
    "          \n",
    "model.add(layers.Dense(units=200, activation='relu'))\n",
    "model.add(layers.Dropout(0.25))\n",
    "\n",
    "model.add(layers.Dense(units=10, activation='softmax'))\n",
    "\n",
    "model.summary()          "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BLdOV8vYgBQS"
   },
   "outputs": [],
   "source": [
    "adam = keras.optimizers.Adam(lr = 0.0001)\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy, \n",
    "              optimizer=adam, \n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 3434
    },
    "colab_type": "code",
    "id": "pxI5Gi4xgBQV",
    "outputId": "9c9b10c7-19bd-4e74-eef1-e1cb482e3256"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 41580 samples, validate on 420 samples\n",
      "Epoch 1/100\n",
      "41580/41580 [==============================] - 7s 162us/sample - loss: 0.7363 - acc: 0.7790 - val_loss: 0.2267 - val_acc: 0.9429\n",
      "Epoch 2/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.2373 - acc: 0.9282 - val_loss: 0.1327 - val_acc: 0.9571\n",
      "Epoch 3/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.1663 - acc: 0.9491 - val_loss: 0.0920 - val_acc: 0.9643\n",
      "Epoch 4/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.1315 - acc: 0.9602 - val_loss: 0.0642 - val_acc: 0.9762\n",
      "Epoch 5/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.1080 - acc: 0.9663 - val_loss: 0.0469 - val_acc: 0.9857\n",
      "Epoch 6/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0927 - acc: 0.9715 - val_loss: 0.0412 - val_acc: 0.9857\n",
      "Epoch 7/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0797 - acc: 0.9759 - val_loss: 0.0294 - val_acc: 0.9905\n",
      "Epoch 8/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0720 - acc: 0.9777 - val_loss: 0.0281 - val_acc: 0.9881\n",
      "Epoch 9/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0640 - acc: 0.9802 - val_loss: 0.0261 - val_acc: 0.9905\n",
      "Epoch 10/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0568 - acc: 0.9823 - val_loss: 0.0227 - val_acc: 0.9929\n",
      "Epoch 11/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0503 - acc: 0.9841 - val_loss: 0.0180 - val_acc: 0.9952\n",
      "Epoch 12/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0463 - acc: 0.9856 - val_loss: 0.0298 - val_acc: 0.9905\n",
      "Epoch 13/100\n",
      "41580/41580 [==============================] - 6s 155us/sample - loss: 0.0420 - acc: 0.9871 - val_loss: 0.0176 - val_acc: 0.9952\n",
      "Epoch 14/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0403 - acc: 0.9873 - val_loss: 0.0120 - val_acc: 0.9976\n",
      "Epoch 15/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0363 - acc: 0.9886 - val_loss: 0.0157 - val_acc: 0.9952\n",
      "Epoch 16/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0333 - acc: 0.9888 - val_loss: 0.0191 - val_acc: 0.9952\n",
      "Epoch 17/100\n",
      "41580/41580 [==============================] - 6s 152us/sample - loss: 0.0298 - acc: 0.9902 - val_loss: 0.0118 - val_acc: 0.9952\n",
      "Epoch 18/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0291 - acc: 0.9904 - val_loss: 0.0092 - val_acc: 0.9976\n",
      "Epoch 19/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0275 - acc: 0.9914 - val_loss: 0.0138 - val_acc: 0.9952\n",
      "Epoch 20/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0240 - acc: 0.9919 - val_loss: 0.0144 - val_acc: 0.9976\n",
      "Epoch 21/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0236 - acc: 0.9928 - val_loss: 0.0095 - val_acc: 0.9976\n",
      "Epoch 22/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0218 - acc: 0.9926 - val_loss: 0.0164 - val_acc: 0.9929\n",
      "Epoch 23/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0196 - acc: 0.9938 - val_loss: 0.0128 - val_acc: 0.9976\n",
      "Epoch 24/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0182 - acc: 0.9940 - val_loss: 0.0084 - val_acc: 1.0000\n",
      "Epoch 25/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0175 - acc: 0.9944 - val_loss: 0.0090 - val_acc: 0.9976\n",
      "Epoch 26/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0167 - acc: 0.9951 - val_loss: 0.0146 - val_acc: 0.9905\n",
      "Epoch 27/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0148 - acc: 0.9951 - val_loss: 0.0099 - val_acc: 0.9952\n",
      "Epoch 28/100\n",
      "41580/41580 [==============================] - 6s 155us/sample - loss: 0.0145 - acc: 0.9953 - val_loss: 0.0111 - val_acc: 0.9976\n",
      "Epoch 29/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0156 - acc: 0.9945 - val_loss: 0.0087 - val_acc: 0.9976\n",
      "Epoch 30/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0129 - acc: 0.9961 - val_loss: 0.0099 - val_acc: 0.9976\n",
      "Epoch 31/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0128 - acc: 0.9959 - val_loss: 0.0062 - val_acc: 0.9976\n",
      "Epoch 32/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0120 - acc: 0.9963 - val_loss: 0.0066 - val_acc: 0.9976\n",
      "Epoch 33/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0107 - acc: 0.9966 - val_loss: 0.0106 - val_acc: 0.9976\n",
      "Epoch 34/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0107 - acc: 0.9965 - val_loss: 0.0208 - val_acc: 0.9929\n",
      "Epoch 35/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0111 - acc: 0.9964 - val_loss: 0.0049 - val_acc: 0.9976\n",
      "Epoch 36/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0101 - acc: 0.9967 - val_loss: 0.0049 - val_acc: 1.0000\n",
      "Epoch 37/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0083 - acc: 0.9974 - val_loss: 0.0129 - val_acc: 0.9976\n",
      "Epoch 38/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0089 - acc: 0.9972 - val_loss: 0.0052 - val_acc: 0.9976\n",
      "Epoch 39/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0079 - acc: 0.9974 - val_loss: 0.0047 - val_acc: 0.9976\n",
      "Epoch 40/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0076 - acc: 0.9976 - val_loss: 0.0057 - val_acc: 0.9976\n",
      "Epoch 41/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0083 - acc: 0.9974 - val_loss: 0.0073 - val_acc: 0.9976\n",
      "Epoch 42/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0078 - acc: 0.9972 - val_loss: 0.0029 - val_acc: 1.0000\n",
      "Epoch 43/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0070 - acc: 0.9978 - val_loss: 0.0133 - val_acc: 0.9976\n",
      "Epoch 44/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0085 - acc: 0.9973 - val_loss: 0.0167 - val_acc: 0.9976\n",
      "Epoch 45/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0065 - acc: 0.9981 - val_loss: 0.0046 - val_acc: 0.9976\n",
      "Epoch 46/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0058 - acc: 0.9981 - val_loss: 0.0081 - val_acc: 0.9976\n",
      "Epoch 47/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0063 - acc: 0.9979 - val_loss: 0.0144 - val_acc: 0.9976\n",
      "Epoch 48/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0061 - acc: 0.9983 - val_loss: 0.0037 - val_acc: 0.9976\n",
      "Epoch 49/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0055 - acc: 0.9982 - val_loss: 0.0122 - val_acc: 0.9929\n",
      "Epoch 50/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0065 - acc: 0.9978 - val_loss: 0.0081 - val_acc: 0.9976\n",
      "Epoch 51/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0071 - acc: 0.9975 - val_loss: 0.0022 - val_acc: 1.0000\n",
      "Epoch 52/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0054 - acc: 0.9982 - val_loss: 0.0030 - val_acc: 1.0000\n",
      "Epoch 53/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0046 - acc: 0.9987 - val_loss: 0.0092 - val_acc: 0.9929\n",
      "Epoch 54/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0053 - acc: 0.9982 - val_loss: 0.0039 - val_acc: 0.9976\n",
      "Epoch 55/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0047 - acc: 0.9985 - val_loss: 0.0027 - val_acc: 1.0000\n",
      "Epoch 56/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0051 - acc: 0.9985 - val_loss: 0.0034 - val_acc: 0.9976\n",
      "Epoch 57/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0049 - acc: 0.9984 - val_loss: 0.0080 - val_acc: 0.9976\n",
      "Epoch 58/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0044 - acc: 0.9987 - val_loss: 0.0088 - val_acc: 0.9952\n",
      "Epoch 59/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0048 - acc: 0.9984 - val_loss: 0.0030 - val_acc: 0.9976\n",
      "Epoch 60/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0041 - acc: 0.9988 - val_loss: 0.0010 - val_acc: 1.0000\n",
      "Epoch 61/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0041 - acc: 0.9988 - val_loss: 0.0029 - val_acc: 0.9976\n",
      "Epoch 62/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0040 - acc: 0.9987 - val_loss: 0.0028 - val_acc: 1.0000\n",
      "Epoch 63/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0040 - acc: 0.9986 - val_loss: 0.0192 - val_acc: 0.9976\n",
      "Epoch 64/100\n",
      "41580/41580 [==============================] - 6s 155us/sample - loss: 0.0037 - acc: 0.9990 - val_loss: 0.0073 - val_acc: 0.9976\n",
      "Epoch 65/100\n",
      "41580/41580 [==============================] - 6s 156us/sample - loss: 0.0040 - acc: 0.9988 - val_loss: 0.0132 - val_acc: 0.9976\n",
      "Epoch 66/100\n",
      "41580/41580 [==============================] - 6s 155us/sample - loss: 0.0050 - acc: 0.9983 - val_loss: 0.0110 - val_acc: 0.9976\n",
      "Epoch 67/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0035 - acc: 0.9987 - val_loss: 0.0084 - val_acc: 0.9976\n",
      "Epoch 68/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0031 - acc: 0.9991 - val_loss: 0.0098 - val_acc: 0.9976\n",
      "Epoch 69/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0037 - acc: 0.9989 - val_loss: 0.0160 - val_acc: 0.9976\n",
      "Epoch 70/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0030 - acc: 0.9991 - val_loss: 0.0022 - val_acc: 1.0000\n",
      "Epoch 71/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0030 - acc: 0.9991 - val_loss: 0.0033 - val_acc: 0.9976\n",
      "Epoch 72/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0035 - acc: 0.9990 - val_loss: 0.0062 - val_acc: 0.9976\n",
      "Epoch 73/100\n",
      "41580/41580 [==============================] - 6s 152us/sample - loss: 0.0032 - acc: 0.9993 - val_loss: 0.0050 - val_acc: 0.9976\n",
      "Epoch 74/100\n",
      "41580/41580 [==============================] - 6s 155us/sample - loss: 0.0037 - acc: 0.9990 - val_loss: 0.0017 - val_acc: 1.0000\n",
      "Epoch 75/100\n",
      "41580/41580 [==============================] - 6s 155us/sample - loss: 0.0031 - acc: 0.9989 - val_loss: 0.0023 - val_acc: 1.0000\n",
      "Epoch 76/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0036 - acc: 0.9988 - val_loss: 0.0088 - val_acc: 0.9976\n",
      "Epoch 77/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0036 - acc: 0.9989 - val_loss: 0.0146 - val_acc: 0.9976\n",
      "Epoch 78/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0023 - acc: 0.9993 - val_loss: 0.0056 - val_acc: 0.9952\n",
      "Epoch 79/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0027 - acc: 0.9991 - val_loss: 0.0159 - val_acc: 0.9976\n",
      "Epoch 80/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0019 - acc: 0.9995 - val_loss: 8.7428e-04 - val_acc: 1.0000\n",
      "Epoch 81/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0033 - acc: 0.9990 - val_loss: 0.0070 - val_acc: 0.9976\n",
      "Epoch 82/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0029 - acc: 0.9993 - val_loss: 7.8337e-04 - val_acc: 1.0000\n",
      "Epoch 83/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0023 - acc: 0.9995 - val_loss: 0.0089 - val_acc: 0.9952\n",
      "Epoch 84/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0032 - acc: 0.9990 - val_loss: 0.0025 - val_acc: 0.9976\n",
      "Epoch 85/100\n",
      "41580/41580 [==============================] - 6s 155us/sample - loss: 0.0030 - acc: 0.9991 - val_loss: 5.2638e-04 - val_acc: 1.0000\n",
      "Epoch 86/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0019 - acc: 0.9994 - val_loss: 0.0019 - val_acc: 0.9976\n",
      "Epoch 87/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0033 - acc: 0.9990 - val_loss: 0.0112 - val_acc: 0.9976\n",
      "Epoch 88/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0032 - acc: 0.9988 - val_loss: 0.0024 - val_acc: 0.9976\n",
      "Epoch 89/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0028 - acc: 0.9991 - val_loss: 1.5998e-04 - val_acc: 1.0000\n",
      "Epoch 90/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0020 - acc: 0.9993 - val_loss: 1.7976e-04 - val_acc: 1.0000\n",
      "Epoch 91/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0020 - acc: 0.9995 - val_loss: 7.8640e-04 - val_acc: 1.0000\n",
      "Epoch 92/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0025 - acc: 0.9993 - val_loss: 3.4278e-04 - val_acc: 1.0000\n",
      "Epoch 93/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0023 - acc: 0.9994 - val_loss: 2.5530e-04 - val_acc: 1.0000\n",
      "Epoch 94/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0025 - acc: 0.9993 - val_loss: 0.0057 - val_acc: 0.9976\n",
      "Epoch 95/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0016 - acc: 0.9996 - val_loss: 2.4696e-04 - val_acc: 1.0000\n",
      "Epoch 96/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0021 - acc: 0.9993 - val_loss: 0.0147 - val_acc: 0.9976\n",
      "Epoch 97/100\n",
      "41580/41580 [==============================] - 6s 153us/sample - loss: 0.0021 - acc: 0.9994 - val_loss: 0.0112 - val_acc: 0.9976\n",
      "Epoch 98/100\n",
      "41580/41580 [==============================] - 6s 155us/sample - loss: 0.0023 - acc: 0.9994 - val_loss: 0.0023 - val_acc: 0.9976\n",
      "Epoch 99/100\n",
      "41580/41580 [==============================] - 6s 154us/sample - loss: 0.0025 - acc: 0.9992 - val_loss: 0.0034 - val_acc: 0.9976\n",
      "Epoch 100/100\n",
      "41580/41580 [==============================] - 6s 155us/sample - loss: 0.0023 - acc: 0.9993 - val_loss: 0.0065 - val_acc: 0.9976\n"
     ]
    }
   ],
   "source": [
    "H = model.fit(x_train, y_train,\n",
    "          batch_size=100,\n",
    "          epochs=100,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CHDGYz_Mki4G"
   },
   "outputs": [],
   "source": [
    "H.history.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 282
    },
    "colab_type": "code",
    "id": "G1s_nCBWk-91",
    "outputId": "bbb8e973-b569-4f0a-d9ae-def57593bf0a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f18421eeef0>]"
      ]
     },
     "execution_count": 65,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4VNX9x/H3rJlMFggkgOCCCB4Q\nlwouLCoo7trSqq2/qq0oVmtRUauttbbV2qp1w71VK61WW3dR68bihqIWFa0gHpBdwpJA9nVm7v39\nMZONsAyBkObm83oenidztznfJHzm5Nxz7/W5rouIiHiLv6MbICIiO5/CXUTEgxTuIiIepHAXEfEg\nhbuIiAcFO7oBDYqKKto8bScvL0pJSfXObE6n0BXr7oo1Q9esWzWnp6Agx7e55Z7ouQeDgY5uQofo\ninV3xZqha9atmneMJ8JdRERaUriLiHiQwl1ExIMU7iIiHqRwFxHxoLSmQhpj9gdeBKZYa+/bZN2x\nwE1AAnjVWntjavkUYATgApOttXN3ZsNFRGTLthnuxpgs4F5g1hY2uQc4AVgNvGOMeQ4oAAZZa0ca\nY4YAU4GRO6fJIiKyLen03OuAk4FfbrrCGDMA2GitXZV6/SowjmS4TwOw1i40xuQZY3KtteU7reWy\nVb6SjUQfuJfYoYdRf/xJHd0cAPxrCsn8y/3Un3QKsRGj2n4g1yXyr8fxFa2n5qeXQEbGzmvkZvhK\nS4jeexf+tWsal9WecSaxo8dtfgfHIfKPv+MrL6fmop9BOLxd7xf8fB4Zzz5NzcQLcfrv3bjcv2I5\nmX99kLrTv0/8W8PaVMtWuS4ZLz5PwH5F9aTJkJ29zV0CC+aTOfUhfLW1yUNEItRMuIDEAQducR/H\ncfH5wOdruvamrrKa4P33Qo+e+Ceej8/vb9iYyL8ex62vp/jMH1NZEwMgv1smoWDTqHLorVmEZ79D\n1cWXUp2TR219nG7ZYQL+zY88+1euIPrAPfgqKpra5bokEi5xx8Hv8xEM+PH7feBCwnFIOC7Vo44k\n9n9nk5ERJJFwKCurIeOxqfhq66g47ydk50bJCAVwAdd1cRyXuJM8rs8H0Ywg4dCumb/vS/d+7saY\n64Hi5sMyxphRwNXW2u+lXk8E9gHygVestS+mls8GJlprF23p+PF4wu2KFy20ixkzYMIEKCxMvj7/\nfJgyBXJzO65NTz0FF18MJSXg88FVV8GNN25/MBcWJut5443k6wMOgMcfhwO3HCZbE4snKK2ox+eD\nQMBHKBggKxJsCp6ZM5Pfy9WrW+yX6N2b/07/D4XlcfwBP316ROnTM4uMNauJXPwTou+/C0DVkP1Z\n9qf7ie07mIxwIPkf24WyqjrKKuuprY+TkxkmNytMyOcQnXIbez50N/5EnFgkkznn/4LFx53O4Lem\ncfjDtxCqqcbxB1hy3qWsOO8SNlTHWbuhmvUl1fj9PrIyQ2RnhujZLZPd8rPom5+F67psLK+jpKKW\n9RurWbuxmnUbqojFHXKyku+dXV3OMY/czNBP3gSgqGdfnjjrV6wYcAChgJ9AwIfrQnlDu6vrOOPz\nf3PGW48RSsRafm8CQT74/k/54rvnkfD7iSccYnGHDWW1rC+pZkNpDQCZGUEyIyEKVi7m4mm3s3fx\ncgA+2/tgnjvnGjICfk7/x00MWTIPgFtOvZr39x0NgN8HBXlReme4nDztAUa//yIAJVndufe4Scwd\ncCjBgI/ePaL07pFFJCNAOBgg6Pexz6xpHPvEHURq23b169y9D+HeEyYRitVzxev3sP/qBQAs7j2Q\nO066gtU9+m11/3DQT2YkSDye/L5kRoLcMXkMvXtE29QeYLNXqO7scL8AGEDrcH8POH9r4b4jtx8o\nKMihqKhi2xt2gPD018BxqT/hpGSobUNgkSU05z1qzzwLMjMbl/vXFJLx9L/w1dUB4Obkkj35ZxTV\nN+uZxONk/e5aog//BTcYpOZnlxF6axahLz4nsedelP/10a32+EJvziT08X/aXKvTM5/as38MkUjT\nwupqcq68lMjzz+BGo1RPmkzGs08RXLaUqoGDKR57AoGAn4Df19hziiWcxh5UPO6Az0fA7yPkxBnw\nytMEy0ooP2Istb370eu5J0gEgswb+z38+T3JzgwR8PkoqayjpKKOypoYiYSD4yZ7UsGAn1Aw+X61\nsQR19QlqwplMP+A4qjOyAIiEA/TODXPOrEcYMetpnECQL8+6mE8OO4GV6ys5+NUnOPWTl/nzMRfy\n6rdObix11KI5XDrjPrLrqvlowKGUZ+Zy3IJZ1AdCvHbQiVSFk/95N+TkM3PoMTj+ps7MbiWFXPna\nXQxeu4ji7J68fuDxfPeTF8muq2ZNt97sVraOqnCUFw4Zzwn/nU5B5QZsn0F80r9tPXi/P/k9jcUd\n/K7DcfNn0rOqhC/7DWFp3305ee5LuD4fsw48jg3RPBpyIhz0kxEOMGTZ5wxc9gWlWd25f9xPWdJr\nHwD6Fy1j0sw/J4/VdzCf7XlQ8nfD7+eDQaOo6D+Qnt0i+IDamnrGvP00p72Z/ID4bMx4sjasZ9D8\nD6iIZONzXbLrqlgwaDgDVyzADQR59A9PUNqzD+s3VpM9fx4XPn87/UoLWZm/J58OGcUpc54jlIjx\nxWHH8023PlTVxqmPJRrrHrh+CYct/ZiqcJS/H/sTvt53GOFggHDIT2Y4SGYkQEY42SuvqUtQUx8n\n6PeREQrQLVHLuCfvZsDCuVRm5RKMx4nUVbPk0KOJZ0Yx775CLJTBR0eMpy6SBb5k6vr9vuR/fRdi\nCYf6mEM84TT+DJzueQy95Vdk5UTblGVbuv3AjoZ7f+Bf1tqRqde/AzaQDPc11toHU8uXAgdZa7fY\naq+Fu6+0hOxfXknkhecAqB1/GpW33omb12PzOzgOmQ8+QNZNN+CrqyO+r6HigYeJH/gtMqY9R/Yv\nrsBfWtpyn1NOoWjqPxs/NKK33UzWbTcn973/IeIHHQz19URvv4XoPXeS6L83Je9/DIGWfyH5KsrJ\n/vUviTz5xA7X/cW403jvot8QDvoJhfyMvv/37PP6M6zf9wBeufB6lmX1Zv2qIsb/+y+c9N83tvv4\ntcEMpo6ZwGsHngg+H8OXfsxlM+6nR1XJDrX76wNG8cQltyV7mOW1HPXK3/jhe/9kVY/dufPEy/m6\nz0Ag+a02GfXcdNs51HbP551/ziARCMLcuZx67bnEgxnM/vGVrDr5+4TDQfrOmcWI+64ns2xji/cr\nGrQ/n157G/X9B7Dbs09w0F9vJVRXy/Kxp7Dsmj+Q268X4bWF7HXd5eR+9D5VI49gwbW3UlmwG25J\nKebO37H7jJd2qObm3FCI8p//irrLLscXDBL6cA45l1xEYOWKLe5Td8p3qLj9bpwePUg4TR/MTnEx\n+dddTbfXWrbPzcig6trfUXPRz/B/s4qcS39K+IP3cQp6UXHXfdQfd2JyyO3RqWT/7tfg91H5+5up\nPedcIv/8BzlXXELs8JGUPvMi0btuJ3r3HeA41Fw0iaprfwuRCIEvF5D7s58Q/HL+FttdM2I0Fff8\nGV///tv/jXIcIlMfIvv3v8UNhqi86VbqzjwLfD7CL08j56rJ+Eu2/3dx4zsfkhiy3/9OuKeWLwBO\nAb4BPgDOJhnuN1hrjzPGDAPusdYesbXjd7pwd11CH84hPsjg5ue3WBWa/U7yP8aaQmLDDwF/gNDc\nj0j07kP15VfhZmW1Olzk6X8Rfu9dnPx86seOI/LsU7jBILHDRxJ+fzZuNErVr35DfP/k8EP0zlsJ\nz36HiptupfaCnxL88AO6f/cknN36UvLW+7jd81ocP/vKS8l8/FFK//IIK446meq6OAA58+Yy8NeX\nEVnzDaWDhrLkwivos3s+WZEQsbjD/KUb+M9X6yitrAeSPRGfL9nLbs7nwk/efoS9i5dz07d/yQeD\nRjJq0Rx+9e9bWVrQn6t+eCuxYHLsOTszRP8+ORwYW09uZSn18WQP3e/3EQ76CYcCjR8O4dRQXSz1\np311/4GsJEpVTYxAwM8evbLZK9Ol57KFlJbXsbGilvqYQ5+eUXbrGSUS3vZppeiU2wm/+xaVf7iF\nmgt/RvA/H9F9/InEe/VmwTPTKQllUVkdo0duhD17ZxMJB8m69mqif32Q8nv+TP2p3yHvmCPwr1hO\n2bMvETtyTMvvTVkpwflfNP7eRJ54jMhzT+NmZhLf/0BCcz/C6d6dylunUPfd01s2znEI2K/occSh\nFG2oarEqsGA+/tId+1BrkNirP87ue7RcWFVF6L+fgeO02t7NzU3+Lm7pr1HXJTD/C/zlZQD4v1lF\n9g3X4S8uJjb8EALW4q+sSH5A3HZXq/9DvnXryM/PpiiQ1Xi8nJ9MIPLSCzgFvfAXrSex+x5U3PsX\nYqOPbPne9fUE532KL95yuAiS5wTiBw+HLYzFp8u/phA3FG7d7pKNBL9csF3HcvJ6kNhvKNC2LGtz\nuBtjhgN3AP2BGMlZMS8By6y1LxhjjgL+lNr8OWvt7an9bgGOAhxgkrX28629T2cKd9+GDeRcNZmM\nV14iPngIJW+83TiEElgwn7wTj4ZEguqrrqH6sivB5yPz/rvJ+tMf8cVa/8I1qDvxFCruuAe3oIDQ\n22+Sc9nFBNauITb8ECruf4jEgIGN2/rXrqHn0aNwKyspe3oaOZMuxF+4mrVPvUTR/sOTf1LWxSmt\nrGNjRR3O4q85/xdnsCp/Dy49Zwquz0/fktXc9fjPCcfreeaw03lqxA+IB0IA9MrLpKYuTkV1jGDA\nz9D+eTgu1NbHcV3onpNBj5wM8nIy6J6dQffsMD1XL8OceSJOOIP5d05l6BXn4a+v550Hn4fBQ+iW\nFaZ7TgY5maEWJ9O2R3v8rP3r1pI3diS+igrKnnmRnEsuwv/NKsqmvUps5OjN77P6G3ocdhCJPfci\nfvBwIs8+RfVlV1J13fVpvWfGi8+TffXl+EtLqR97DBV3P4CzW98tbv+/+Nfp9vIVFZHz88vIeP0V\nnOwcKm++jbof/HCLHxCb1uwrKyXv6NEEvllF7ZlnUfnHP+HmdttVzd8lOqTn3t46S7iHZ75BzuRJ\n+IvW4+QX4C8uombCRCpvnQLV1eSdMJag/Yqyx56k/sSTW+wbWLyI4BbGtJ0+uxEbe0yLX3RfaQmh\nuR9Rf/SxEGzqgcbiDoXFVew1bzYF5/4frt+Pz3GYccK53Dv0e2zpG/nz16YwduE7PHvprWwYMYbT\nrptAwbKFzLriFtaOO5VIRpCyyjoWry5jyepyfMDRw/px7CF70C0rvRkfkUenknP15Y1tqrjjHmp/\nNCGtfdPRXj/r8IzX6Xb2DxrbXXXl1VRf85ut7pN9xSVkPvEYALGDh1H67xkQCqX9nr516wgu+CI5\n62YbH3ZeCHcg+Rfvu2+TGLQvTt+tn3jcXM3+wtX4v/mG+GGHt2crO4zCfRO75Be/spLs668j87Gp\nuOEwVdf8hprzLiDv5HEEF35J2d//SfjtWWT+/RFqJl5I5c2375S3dV2Xkoo6VhdXsXJdBV+tKGHx\nN2XUx5N/Kl/45kN8+7NX+bLvEH71gz8wYM8e9O2ZRWZGgMxwkNzsMD1yIvTIzaDvuuX0PmYUsWHD\niR0+iuif76XmrB9Redf9rd63YdjFv709bNcl9/wfkfHKS9SdOp7yRx5L60RyutrzZ531618Qffgv\nxA45jNKXXm/xgbo5/qVL6DFqOG5mlJI338PZe0C7tAs8FO7bQTWnvY/CfZtqalrMUGkQnPsRuZMu\nJLB8GfEhQyl/4GESQ/cHIPDVQvKOHwP+AL7qKuJDhlLyxlstZ4xsB9d12VBeyxdLN/LFkg3YVSXU\n1CVabNOvIItBu3cnMxKitKgUM+cN4iecyEEjhtCz29bfN3fC2WS8+jIA8X0GUjLj3bTmM28PX2UF\n4ZdfpP7b43Gzc3bqsdv1P3xdHRkvPEv9sSe0GkvdkvD013C692j3nqSCrmtQuG9iZ/wSRJ54jOyf\nX0btWT+i6vc3JUOpvp7oHbcQvftOcF1qJk2m6pe/bjU3O/K3v5LzyytxIxFKZrxLwgxO6z0d12Xh\nihLenreaxatKqYs51McSLYZVeuVlsmfvHPr2jLJ7QTaD9ujeOETSlrqDn88j77gxuKEQpa/NIn7g\nt7Zr/47WFf/DQ9esWzWnvc9mw/1/5jF7HSlgvyL72qvxOQ6Zjz+avNLtmuvIfODexjniFfc9uMWr\nKmsnTMQXqye+7+AtBnvCcViwbCMr1lZQVRunui7OopWlrE9d0NEzN0JeboSMoJ/saJghe+VxwD49\n6dW99V8SOyJ+0MFU3HwbTu/dOl2wi0j6FO61teRedD6+mhrKH/47wS/+S+Z9d5F78QUA1Jz9Y6pu\nvHnrwws+HzUX/qzxZcJxqKiOUV0bp6o2xhdLN/L+F2soqahrsVso6Gf0AX0Ye3A/BuyW2+YZJNur\nduJFu+R9RKTjdPlwz7rxtwS/nE/NuROpG39a8t9xJxK9/25qf3gO9SedkvaxiktreOuz1cz+fE3j\nPTAaZGYEOPrgfhw0MJ+caIhoJEj3rAwywrrlgojsfF0u3MMz3yDr+usgHgfXJbhsKXEzmMob/ti4\nTfzwEZQfPiKt47mui11ZyoyPV/HZ4mJckhfpHDq4F1mRIJmRIP3ysxhuepGxi24YJCLS5cI9eved\nBBdZEr37ABAfegDl9z8E0e27aU/Ccfjoy3VM/88qVq6vBGDv3XIYN3x3Dh3ci5BugiYiHahLhbt/\nxXJCH31A/ZFjKHvu5TYdw3Fd5i5cz7T3lrFuYzU+HxwyuBfHH7oHA/t562o5Eem8ulS4R557Gkje\ni7stFn9TyuPTF7FqfSUBv4+x3+rLySP2In8nz2gREdlRXSfcXZeMZ57EjUSoP/U727VrZU2MZ99e\nwrufJ++PPnJoH8Yf0Z9eeW2+/7KISLvqMuEe/OxTgku+pva7p+HmpP/Qis+/LuZvry6kvDpGv4Is\nzj1hMAN31/CLiPxv6zLhnvHMkwDUff//0to+Fnd45u2vmfnxNwQDfs4Yuw/HH7oHwcCO3SpURGRX\n6BrhHosRmfZc473St2XRqlL+OWMRK9dXslvPKBd9Zyh79t6590gREWlPXSLcw2/Pwl9cTPUFF231\nlqxLVpcxbfZSFixPPgDhqIN244fj9tWFRiLS6XSJcM+Y9jwAdaf/YIvbvP3Zah573QKwX/88vnvk\nAE1tFJFOy/vh7jiE35xBoncf4sMO2ewmSwrLeGL6IrIzQ0z63v6YPfM2u52ISGfh+bODwc8+xb9h\nA/XHHr/Zh0aUV9XzwAvzcVyXn44fqmAXEU/wfLiHZ04HoH7c8a3WJRyHv7w4n5KKOk47agD79e+x\nq5snItIuvB/us6bjBoPExoxtte6VOSv4amUpBw/K5+QRe+36xomItBNPh7tv/XpC8z4lNmJUqwuX\nVq2v5OU5y8nLyWDiKUN22b3URUR2BU+He/itmUDrIZmE4zD11YUkHJdzTxxMNJL+E+tFRDoDb4f7\nrNR4+7Etw/31j1ayYm0Fo/fvw4H79OyIpomItCvvhns8TvitN0nssSeJfU3j4sLiKl58bzndssKc\nOW5QBzZQRKT9eDbcgx/PxV9WSv244xqnQLquyxMzFhFPOPzoBEN2poZjRMSbPBvuGZsZkvns62IW\nrijhgAE9GbZvQUc1TUSk3Xkz3GtryfjX4zjZOdSPPgqAeMLh6Te/xu/zceYxAzu4gSIi7cuT4R75\n5z8IrF9H7XkXQFYWAG9+upp1JTWMPbgvffOzOriFIiLty3vhHosRvf9u3EiE6osmAcknKb303jIy\nM4KMP2LvDm6giEj781y4Zzz3NIFVK6k551zcXr0AeO2jFVTXxfnO6P7kRMMd3EIRkfbnrXBPJIje\ndTtuKETNpMlAcobM3IXriYQDHDNs9w5uoIjIruGpcM94eRrBpUuoPfMsnH7JIP+mqIrisloOGNCT\nUNBT5YqIbJGn0i4y9WFcv5/qSy5vXDZvcREABw/K76hmiYjsct4J95oaQp/MJX7gQTgD9mlcPG9x\nMQG/T7cZEJEuxTPhHvr0Y3yxGLHDRzUu21hey4q1Fey7R3fdHExEuhTvhPuHcwCIjRzduOyzr4sB\nDcmISNeT1jNUjTFTgBGAC0y21s5ttm48cB1QBzxprb3PGDMWeAZYkNrsC2vtpTuz4ZsKfZAK98NH\nNi6btzgZ7t9SuItIF7PNcDfGjAEGWWtHGmOGAFOBkal1fuA+YBiwAXjNGDMttes71toz2qfZm4jF\nCH38H+JmMG7P5Nh6dW2cr1aUsGevbPK7Ze6SZoiI/K9IZ1hmHDANwFq7EMgzxjQ81igfKLXWFllr\nHWAWcGy7tHRr5s3DV11FbETTkMz8ZRtIOK567SLSJaUzLNMH+KTZ66LUsvLU1znGmEHAcuBo4O3U\n1/sZY14CegA3WGtnbO1N8vKiBIOB7Wx+yqPvApB5/DFkFuQAYGcsAuCYw/aiILXMi7xc25Z0xZqh\na9atmtsurTH3TTQ+bNRa6xpjziU5VFMGLEutXwzcADwNDADeMsYMtNbWb+mgJSXVbWhKUsHs2QBs\n2O9gnKIKABavLCEjFCA77KcotcxrCgpyPFvblnTFmqFr1q2a099nc9IJ90KSPfUGfYE1DS+ste8A\nRwIYY24GlltrVwNPpTZZYoxZC/QjGf47l+PA7Nkk9tyr8arUhOOwdmM1uxdk49eDr0WkC0pnzH06\ncAaAMWYYUGitbfxoMca8ZozpZYzJAr4NzDTGnG2MuSq1vg/QG1i901sPBOxXUFJCbETT/Pai0lri\nCVe39hWRLmub4W6tnQN8YoyZA9wDTDLGTDDGfC+1ycMkPwDeA2621hYDLwFjjDGzgReBi7c2JLMj\nQh+8D9Ai3AuLqwAU7iLSZaU15m6tvWaTRZ83W/c88Pwm21eQ7MW3u9CHqXAfqXAXEWnQ6a9Q9W8s\ngQEDSAxoenRe4QaFu4h0bW2ZLfM/pfyRR8nPi4LTdOK0sKiKcNBPfm6kA1smItJxOn3P3e3WHXo2\n3fHRcVzWbKymT88ofr9myohI19Tpw31TxWU1xOKOhmREpEvzXLgXFicvhurbU+EuIl2X98I9dTK1\nn3ruItKFeS/cNQ1SRMSb4R4M+MnvrpkyItJ1eSrcHdelcEMVfXpECfg9VZqIyHbxVAJuLKulPubQ\nNz/a0U0REelQngp3XZkqIpLkrXDXNEgREcBz4Z6aBlmgcBeRrs1T4b6+tAaAgu56ILaIdG2eCvdY\n3CEY8BMMeKosEZHt5qkUTCQcggHdLExExFPhHndc9dpFRPBauCccAuq5i4h4K9wTCYegrkwVEfFW\nuCeHZdRzFxHxVLgnEhpzFxEBj4W7xtxFRJI8Fu6u7gYpIoKHwt11Xc1zFxFJ8Uy4O66LCxpzFxHB\nQ+EeT7gAGnMXEcFD4Z5IOACa5y4igofCvaHnrjF3EREPhXvCaQh3z5QkItJmnknCeGpYRmPuIiIe\nDHf13EVEPBTuiYYxd51QFRHxTrjHHQ3LiIg08E64a567iEgjz4S75rmLiDTxTBJqnruISJNgOhsZ\nY6YAIwAXmGytndts3XjgOqAOeNJae9+29mkPmi0jItJkm0lojBkDDLLWjgQmAvc0W+cH7gNOBo4C\nvm2M2X1r+7SXpjF3hbuISDpJOA6YBmCtXQjkGWNyU+vygVJrbZG11gFmAcduY592kXAaeu4alhER\nSWdYpg/wSbPXRall5amvc4wxg4DlwNHA29vYZ7Py8qIEg4HtaHpL0awMALp3i1JQkNPm43Q2XanW\nBl2xZuiadavmtktrzH0TjV1ja61rjDkXmAqUAcuar9/cPltSUlLdhqYkFRTkNO5fU11HUVFFm4/V\nmRQU5HSZWht0xZqha9atmtPfZ3PSCfdCkr3uBn2BNQ0vrLXvAEcCGGNuJtmDj2xtn/YQ143DREQa\npZOE04EzAIwxw4BCa23jR4sx5jVjTC9jTBbwbWDmtvZpD403DvNrzF1EZJs9d2vtHGPMJ8aYOYAD\nTDLGTADKrLUvAA+TDHMXuNlaWwwUb7pPu1WQ0nhvGfXcRUTSG3O31l6zyaLPm617Hng+jX3aVdM8\nd/XcRUQ8081tup+7Z0oSEWkzzyRh05OY1HMXEfFMuOv2AyIiTTyThA0nVDVbRkTEQ+Guee4iIk08\nk4SaLSMi0sQz4Z7QmLuISCPPJGFcY+4iIo08FO6a5y4i0sAzSah57iIiTTwT7prnLiLSxDNJqDF3\nEZEmngn3hOMQ8Pvw+RTuIiKeCfd4wtWQjIhIimfSMJFwdDJVRCTFM+EeT7iaBikikuKZNIwnHJ1M\nFRFJ8Uy4JxxXwzIiIimeCfd4wtEJVRGRFM+kYTzhEvB7phwRkR3imTTUbBkRkSaeCXfNcxcRaeKJ\nNHQcF8fVCVURkQaeCPeEo9v9iog054k0jMVTd4TUPHcREcAj4d5wR0iNuYuIJHkiDZuewqSeu4gI\neCXcU8MymucuIpLkiTRsegqTeu4iIuCRcI/pEXsiIi14Ig0bh2XUcxcRAbwS7uq5i4i04Ik0bJoK\nqZ67iAh4JtwbLmLyRDkiIjvME2moee4iIi0F09nIGDMFGAG4wGRr7dxm6yYB5wAJ4GNr7eXGmAnA\njcCS1GYzrLV/3JkNb67hhKrG3EVEkrYZ7saYMcAga+1IY8wQYCowMrUuF7gaGGitjRtjphtjRqR2\nfcpae1V7Nby5hjF3PUNVRCQpna7uOGAagLV2IZCXCnWA+tS/bGNMEIgCG9ujoVujee4iIi2lMyzT\nB/ik2eui1LJya22tMeYGYClQAzxprV1kjBkFjDHGvA6EgKustfO29iZ5eVGCwUCbiogvK0keo3sm\nBQU5bTpGZ9XV6oWuWTN0zbpVc9ulNea+icaxj1QP/lpgX6AceNMYcxDwIVBkrX3FGDMSeAw4YGsH\nLSmpbkNTkhpOqFZX11NUVNHm43Q2BQU5Xape6Jo1Q9esWzWnv8/mpDOOUUiyp96gL7Am9fUQYKm1\ntthaWw/MBoZba7+y1r4CYK39ACgwxrStW54GXcQkItJSOmk4HTgDwBgzDCi01jZ8tCwHhhhjMlOv\nDwEWG2N+YYz5YWqf/Un24hOYRLydAAAITklEQVQ7teXNNF7EpBOqIiJAGsMy1to5xphPjDFzAAeY\nlJrqWGatfcEYcxvwljEmDsyx1s42xiwD/mGM+WnqPSa2Yw3N5rmr5y4iAmmOuVtrr9lk0efN1j0I\nPLjJ9t8AR+9w69KkW/6KiLTkia6uLmISEWnJE2mo2w+IiLTkiXCP6cZhIiIteCIN9bAOEZGWvBHu\njfdz90Q5IiI7zBNp2HQ/d/XcRUTAK+Ee1zx3EZHmPJGGcUfz3EVEmvNGuMc15i4i0pwn0lBXqIqI\ntOSpcA9onruICOCRcI/FHfw+H37NlhERATwS7vGEowuYRESa8Uy4a7xdRKSJZ8Jd4+0iIk08kYjx\nuKueu4hIM94Id8fRHHcRkWY8kYjxuKNbD4iINOOJRNQJVRGRlrwT7jqhKiLSyBOJGNMJVRGRFjp9\nuLuum7qIqdOXIiKy03T6REw4yTtCBnTrARGRRp0/3PWIPRGRVjp9IupBHSIirXX+cE/13DXmLiLS\npNMnYkIP6hARaaXTh3s8dUJV89xFRJp0+kRUz11EpLVOH+4acxcRaa3TJ6Ieji0i0lqnD3fNcxcR\naa3TJ2JDz11XqIqINOn84Z66iElj7iIiTTp9IsYbh2XUcxcRadDpw71xKqTmuYuINAqms5ExZgow\nAnCBydbauc3WTQLOARLAx9bay40xIeDvwF6p5edZa5fu5LYDTXeFVM9dRKTJNru7xpgxwCBr7Uhg\nInBPs3W5wNXAkdbaI4D9jDEjgLOA0tSyPwI3t0fjodkJVY25i4g0SicRxwHTAKy1C4G8VKgD1Kf+\nZRtjgkAU2Jja54XUNjOB0Tuz0c1pzF1EpLV0hmX6AJ80e12UWlZura01xtwALAVqgCettYuMMX1S\n22GtdYwxrjEmbK2t39Kb5OVFCQYD211AZjQMQI/uWRQU5Gz3/p2dau46umLdqrnt0hpz30RjFznV\ng78W2BcoB940xhy0tX22pKSkug1NgdLSGgCqquooKqpo0zE6q4KCHNXcRXTFulVz+vtsTjrDMoUk\ne+oN+gJrUl8PAZZaa4tTvfLZwPDm+6ROrvq21mvfEU3z3DUsIyLSIJ1wnw6cAWCMGQYUWmsbPlqW\nA0OMMZmp14cAi1P7fD+17NvAWzurwZvSmLuISGvbHJax1s4xxnxijJkDOMAkY8wEoMxa+4Ix5jbg\nLWNMHJhjrZ1tjAkAxxlj3gPqgAntVYDmuYuItJbWmLu19ppNFn3ebN2DwIObbJ8Aztvh1qUhrhuH\niYi00ukTMaExdxGRVjp9uKvnLiLSWqdPRD1mT0SktU4f7nrMnohIa50+ERvmuQf1sA4RkUadP9w1\n5i4i0kqnT8REQrNlREQ21enDvbHnrouYREQadfpEjKvnLiLSSqcP98ZhGZ1QFRFp1PnD3XEJBvz4\nfAp3EZEGnT7c4wmXUFDBLiLSXOcPd8choJOpIiIttOVJTP9TjjywL67G20VEWuj04X78oXt0ycdx\niYhsjcYzREQ8SOEuIuJBCncREQ9SuIuIeJDCXUTEgxTuIiIepHAXEfEghbuIiAf5XNft6DaIiMhO\npp67iIgHKdxFRDxI4S4i4kEKdxERD1K4i4h4kMJdRMSDFO4iIh7U6R/WYYyZAowAXGCytXZuBzep\nXRhjbgWOJPkzuxmYC/wDCABrgB9Za+s6roXtxxiTCcwHbgRm4fG6jTFnA78A4sBvgf/i/ZqzgceA\nPCADuAFYC/yZ5P/t/1prL+64Fu5cxpj9gReBKdba+4wxe7CZn3Hqd+FywAEestY+ku57dOqeuzFm\nDDDIWjsSmAjc08FNahfGmKOB/VN1ngjcBfweuN9aeyTwNXB+BzaxvV0HbEx97em6jTE9gd8BRwCn\nAuPxeM0pEwBrrT0aOAO4m+Tv+WRr7WigmzHmpA5s305jjMkC7iXZUWnQ6mec2u63wLHAWOAKY0yP\ndN+nU4c7MA6YBmCtXQjkGWNyO7ZJ7eJd4Pupr0uBLJI/7JdSy14m+QvgOcaYwcB+wCupRWPxdt3H\nAjOttRXW2jXW2gvxfs0AxUDP1Nd5JD/M9272l7iX6q4DTgYKmy0bS+uf8eHAXGttmbW2BngfGJ3u\nm3T2cO8DFDV7XZRa5inW2oS1tir1ciLwKpDV7E/z9cBuHdK49ncHcGWz116vuz8QNca8ZIyZbYwZ\nh/drxlr7JLCnMeZrkp2Zq4CSZpt4pm5rbTwV1s1t7me8ab5t1/egs4f7pnwd3YD2ZIwZTzLcL9lk\nlSfrNsb8GPjAWrtsC5t4sW4fyR7saSSHKv5Gyzq9WDPGmHOAldbagcAxwOObbOLJurdgS7Vu1/eg\ns4d7IS176n1JnozwHGPMCcCvgZOstWVAZepEI0A/Wv6J5xWnAOONMR8CFwC/wft1rwPmpHp3S4AK\noMLjNUNyuOENAGvt50AmkN9svVfrbrC53+tN8227vgedPdynkzz5gjFmGFBora3o2CbtfMaYbsBt\nwKnW2oYTizOB01Nfnw683hFta0/W2jOttYdaa0cAfyU5W8brdU8HjjHG+FMnV7Pxfs2QPIl4OIAx\nZi+SH2oLjTFHpNafhjfrbrC5n/FHwKHGmO6p2USjgdnpHrDT3/LXGHMLcBTJqUKTUp/6nmKMuRC4\nHljUbPG5JAMvAqwAzrPWxnZ963YNY8z1wHKSvbvH8HDdxpiLSA6/AfyB5LRXr9ecDUwFepOc7vsb\nklMhHyTZCf3IWnvllo/QeRhjhpM8l9QfiAGrgbOBv7PJz9gYcwZwNcnpoPdaa59I9306fbiLiEhr\nnX1YRkRENkPhLiLiQQp3EREPUriLiHiQwl1ExIMU7iIiHqRwFxHxoP8HyQgqFwkqR0wAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(H.history['acc'])\n",
    "plt.plot(H.history['val_acc'],'r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 282
    },
    "colab_type": "code",
    "id": "-AGE67k2lGxL",
    "outputId": "3919dfd2-25e5-4ff6-c9ed-6cd31a6b31e1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f184220f4a8>]"
      ]
     },
     "execution_count": 66,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8XHW9//HXnNmSyd5k2nTfKF9a\naKFgoQW70CI78gOLoly1Ater4r1cvXJFvS7ggoqI4HVDNlEuoAi1bFKFgsW2UAq0dPuWFrrRLW3T\n7JPJLL8/ZpJm7zQkDSfzfj4ePMicc+bM95Ok7/nmO99zvp5kMomIiLiX098NEBGR90ZBLiLicgpy\nERGXU5CLiLicglxExOV8x/oFKypqejxNpqQkRGVlfW82xxWyse5srBmys+5srBmOvu5wuMDT1T5X\n9ch9Pm9/N6FfZGPd2VgzZGfd2Vgz9G7drgpyERHpSEEuIuJyCnIREZdTkIuIuJyCXETE5RTkIiIu\npyAXEXE51wT5weoI9z+5jsZovL+bIiLyvuKaIH914z7+vGQzdkdlfzdFROR9xTVB7nFSV6c2xRL9\n3BIRkfcX1wS5z5tqaiyuFY1ERFpzTZB70z3yWFw9chGR1lwT5D5vKsjjCfXIRURac1GQNw+tqEcu\nItKaa4Lc66SaGtcYuYhIGxktLGGMuR2YDiSB6621K9PbhwMPtjp0HHCjtfb/er2h6aGVWEI9chGR\n1o4Y5MaY2cAEa+0MY8xE4F5gBoC19l1gTvo4H/ACsKhPGqpZKyIincpkaGUesBDAWrsBKDHGFHZy\n3ALgz9ba2t5r3mEtH3ZqjFxEpI1MhlbKgVWtHlekt1W3O+5a4NwjnaykJNSjJY4O1DUBEAj6CYcL\njvr5bqeas0c21p2NNUPv1d2TxZc7LABqjJkBbLTWtg/3Dnq6yGp1dUPq/zURKipqenQOtwqHC1Rz\nlsjGurOxZjj6ursL/UyGVnaR6oE3GwbsbnfMxcDfM25RDzSPkWvWiohIW5kE+WJgPoAx5lRgl7W2\n/dvINGB1L7etDc1aERHp3BGD3Fq7DFhljFkG3AlcZ4xZYIy5rNVhQ4F9fdRGALzqkYuIdCqjMXJr\n7Y3tNq1ut39yr7WoCz5HPXIRkc645spOzSMXEemci4Jc88hFRDrjmiD3qkcuItIp9wS57kcuItIp\n1wW5hlZERNpyTZB7PB58XoeYFpYQEWnDNUEO4Pd5NI9cRKQdVwV5qkeuoRURkdbcF+TqkYuItOGu\nIPc5+rBTRKQddwW519H0QxGRdlwW5B4NrYiItOOyIHeI68NOEZE2XBfk6pGLiLTluiDXPHIRkbZc\nFeR+n0MimSShqztFRFq4Kshb1u3UOLmISAtXBrnGyUVEDstoqTdjzO3AdCAJXG+tXdlq30jgISAA\nvGat/VxfNBTA59OtbEVE2jtij9wYMxuYYK2dAVxDagHm1m4DbrPWng7EjTGjer+ZKT5HPXIRkfYy\nGVqZBywEsNZuAEqMMYUAxhgHmAksSu+/zlq7vY/ais+XHiNXj1xEpEUmQyvlwKpWjyvS26qBMFAD\n3G6MORVYaq39WncnKykJ4fN5e9bY9Bh5YXGIcDi/R+dwq3C4oL+bcMxlY82QnXVnY83Qe3VnNEbe\njqfd18OBO4CtwFPGmIustU919eTKyvoevGRK8wLMFRU1BMie4ZVwuICKipr+bsYxlY01Q3bWnY01\nw9HX3V3oZzK0sotUD7zZMGB3+uv9wDZr7RZrbRx4Djgx45YdpeahFY2Ri4gclkmQLwbmA6SHT3ZZ\na2sArLUx4G1jzIT0sacBti8aCuBvnn6oeeQiIi2OOLRirV1mjFlljFkGJIDrjDELgCpr7ePAfwL3\npz/4fBN4os8a23xBkHrkIiItMhojt9be2G7T6lb7NgMf7M1GdcXbckGQeuQiIs1cdmVn8wVB6pGL\niDRzVZD7NY9cRKQDVwV5y71WdPdDEZEWrgxy9chFRA5zZZBrjFxE5DB3BblP88hFRNpzVZD7NY9c\nRKQDVwW516v7kYuItOeqIPfpgiARkQ5cFeQaWhER6chVQa4PO0VEOnJXkKfHyNUjFxE5zGVBrnnk\nIiLtuSvINbQiItKBq4Lcr0v0RUQ6cFWQezW0IiLSgauC3KcLgkREOnBZkGseuYhIexkt9WaMuR2Y\nDiSB6621K1vt2wrsAOLpTVdZa9/t3WamNC8soR65iMhhRwxyY8xsYIK1doYxZiJwLzCj3WEXWGtr\n+6KBrbX0yLWwhIhIi0yGVuYBCwGstRuAEmNMYZ+2qgtafFlEpKNMhlbKgVWtHlekt1W32vZrY8wY\n4CXga9baLrvMJSUhfD5vD5qa4jgePI5DOFzQ43O4UbbVC9lZM2Rn3dlYM/Re3RmNkbfjaff4W8Bf\ngYOkeu4fAR7t6smVlfU9eMmUcLgAn+Mh0thERUVNj8/jNuFwQVbVC9lZM2Rn3dlYMxx93d2FfiZB\nvotUD7zZMGB38wNr7QPNXxtjngYm002Qv1der0fzyEVEWslkjHwxMB/AGHMqsMtaW5N+XGSMedYY\nE0gfOxtY2yctTfM6jsbIRURaOWKP3Fq7zBizyhizDEgA1xljFgBV1trH073wFcaYBuB1+rA3DqmL\ngjSPXETksIzGyK21N7bbtLrVvjuAO3qzUd3xeR3dNEtEpBVXXdkJqSmIGiMXETnMdUGeGlpRj1xE\npJn7gtxxiOnKThGRFu4LcvXIRUTacF2Qe53UPPJkUr1yERFwY5DrxlkiIm24Lsh1T3IRkbZcGOTp\nVYI0l1xEBHBhkGvdThGRtlwX5M09cs1cERFJcV+QO+keuT7sFBEB3Bjk6pGLiLThuiD3OhojFxFp\nzX1B3jxrRT1yERHAhUGueeQiIm25MMjVIxcRac11Qd4yj1wXBImIABmuEGSMuR2YDiSB6621Kzs5\n5hZghrV2Tq+2sJ3Ds1Y0tCIiAhn0yI0xs4EJ1toZwDXAnZ0cMwmY1fvN68inWSsiIm1kMrQyD1gI\nYK3dAJQYYwrbHXMb8I1eblunWnrkGloREQEyC/JyoKLV44r0NgCMMQuAF4Gtvdmwrhy+14qCXEQE\nMhwjb8fT/IUxZhDwGeAcYHgmTy4pCeHzeXvwsunnF+cCkBsKEg4X9Pg8bpNNtTbLxpohO+vOxpqh\n9+rOJMh30aoHDgwDdqe/nguEgaVAEBhvjLndWvulrk5WWVnfw6amiq6viwJw6FA9FRU1PT6Xm4TD\nBVlTa7NsrBmys+5srBmOvu7uQj+ToZXFwHwAY8ypwC5rbQ2AtfZRa+0ka+104DLgte5CvDccnkeu\nDztFRCCDILfWLgNWGWOWkZqxcp0xZoEx5rI+b10nNI9cRKStjMbIrbU3ttu0upNjtgJz3nuTuqd5\n5CIibbnuys7D88jVIxcRATcGefNNs7SwhIgI4MIg121sRUTacl+QO5q1IiLSmuuC/PD9yNUjFxEB\nVwa5euQiIq25MMg1j1xEpDXXBblXS72JiLThuiDXUm8iIm25L8gdzSMXEWnNdUGueeQiIm25L8g1\nj1xEpA3XBbnH48HreDSPXEQkzXVBDqkpiOqRi4ikuDTIPZpHLiKS5sog93odzSMXEUlzZZD7vB7N\nWhERSXNlkHsdj+aRi4ikZbTUmzHmdmA6kASut9aubLXvX4FrgDipJeCus9b2acr6vA6RaFNfvoSI\niGscsUdujJkNTLDWziAV2He22hcCrgRmWmvPAk4AZvRRW1t4Hc1aERFplsnQyjxgIYC1dgNQYowp\nTD+ut9bOs9Y2pUO9CNjTZ61N83k1j1xEpFkmQyvlwKpWjyvS26qbNxhjbgSuB35mrX27u5OVlITw\n+bw9aGpKOFxAbo6fWCJJOFzQ4/O4TTbV2iwba4bsrDsba4beqzujMfJ2PO03WGt/aIy5A3jaGPOS\ntfafXT25srK+By+ZEg4XUFFRQzKRIJFIsndfNY6nQ3MGnOa6s0k21gzZWXc21gxHX3d3oZ/J0Mou\nUj3wZsOA3QDGmEHGmFkA1toG4BngrIxb1kO6J7mIyGGZBPliYD6AMeZUYJe1tvltxA/cb4zJTz8+\nHbC93sp2fI7ugCgi0uyIQyvW2mXGmFXGmGVAArjOGLMAqLLWPm6MuRlYYoyJkZp+uKhPW0yrHrnm\nkouIZDZGbq29sd2m1a323Q/c33tNOjKtEiQicphLr+xML8CsIBcRcWeQN/fI9WGniIhrg1w9chGR\nZq4M8sPrdqpHLiLiyiD3adaKiEgLlwa5Zq2IiDRzZZA3z1rRjbNERFwa5C09cg2tiIi4M8g1j1xE\n5DBXBrnmkYuIHObSIFePXESkmSuDPCeYWpiitkHrdoqIuDLIhw7KA2D3wZ4vUiEiMlC4Jsi9G9bD\nZZfhOXCA8tIQHmD3/rr+bpaISL9zTZD7X14OCxcSePF5gn4vpUU57DqgHrmIiGuCPDF8OADOzh0A\nDC3No7ouSl1E4+Qikt1cE+TxEaMA8O5IBfmwshAAu/erVy4i2c01QZ4YORIAZ+d2INUjB9h1QOPk\nIpLdMlrqzRhzOzAdSALXW2tXttp3NnALECe18PK11tpen+CdzC+AkhK8O1JBPqw5yPWBp4hkuSP2\nyI0xs4EJ1toZwDXAne0OuQuYb609CygAzu/1VjYbPRrvzh2QTDK0eWhFH3iKSJbLZGhlHrAQwFq7\nASgxxhS22n+atXZn+usKoLR3m9jKmDF46uvxHDxIXo6forwAuzW0IiJZLpOhlXJgVavHFelt1QDW\n2moAY8xQ4Fzgm92drKQkhM/n7VFjGT0agLK6A3DCGEYPLeTNLfspKMwlJ5jRKJFrhcMF/d2EYy4b\na4bsrDsba4beq7sn6edpv8EYMxh4AviCtfZAd0+urOz5UEg4HeRVazYSHXU8pYVBkklYu2kfo8sH\n7i9COFxARUVNfzfjmMrGmiE7687GmuHo6+4u9DMZWtlFqgfebBiwu/lBepjlGeB/rLWLM25VT6SD\nvMMHnhpeEZEslkmQLwbmAxhjTgV2WWtbv43cBtxurf1rH7SvrXSQN09BHFba/IGnglxEstcRh1as\ntcuMMauMMcuABHCdMWYBUAU8C3wKmGCMuTb9lP+z1t7VJ60dMwY4fFHQ0LL0zbN0UZCIZLGMxsit\ntTe227S61dfB3mvOEQwaRDKU13KZflFegNygT0MrIpLVXHNlJwAeD/GRI/Gmh1Y8Hg/DykLsq2zQ\nIhMikrXcFeRAfMRInEOH8NRUA6lL9eOJJPsqG/q5ZSIi/cN1QZ5I3zzLab55Vnrmij7wFJFs5bog\nj49M3wUxPbwycnA+AG/trOq3NomI9CfXBXnLXRDTPfLjRxaTG/Syyu4jmUz2Z9NERPqF64I8PiIV\n5N70zBW/z+GU48o4UN3I1j3Zd3WYiIjrgjwxsnmMfHvLtg+YwQC8unFfv7RJRKQ/uS/IBw8hGQi0\njJEDnDRuEMGAl1c1vCIiWch1QY7jEB8+ouXqTgC/z8spx5VRcSjC9r21/dg4EZFjz31BTmoKolOx\nDyKRlm0fMGEAXrUaXhGR7OLKII+nZ6543z3cKz9pXCkBv8PKjRpeEZHs4sogT6RnrjjbD4+TB/1e\npowvY19lAzsrdHGQiGQPVwZ5fOw4AHzr17XZ3jy8smzt7g7PEREZqFwZ5E0fnAVAYMlzbbZPnVBG\ncX6AF17fRW1DU380TUTkmHNlkCeGlNN00hT8K/4JdYeHUfw+L+efMZrGpjiLV+7o5gwiIgOHK4Mc\noGnuOXiiUQLLlrbZPvuUYRSG/Dy3agf1EfXKRWTgc22QR+d9CIDA839vsz3o93Le6aNoaIzz91U7\n+6NpIiLHlGuDvOkDp5PIL8DfLsgB5kwdTl6Oj7+t3EFDY6wfWicicuxkFOTGmNuNMcuNMcuMMdPa\n7csxxvzOGPNq3zSxC34/TbPm4HvnbZy3t7TZlRv0ce60kdRFYjz7yvYuTiAiMjAcMciNMbOBCdba\nGcA1wJ3tDrkVeKMP2nZE0bnnAB1nrwDMO20kxfkBnly2jU07Dh3rpomIHDOZ9MjnAQsBrLUbgBJj\nTGGr/V8HHu+Dth1R9Ox5AASWdBxeCeX4+NylJ5EkyW8WraOmPnqsmycickz4MjimHFjV6nFFels1\ngLW2xhhTmukLlpSE8Pm8R9XI1sLhglYPToSJEwm+9A/ChQEIBjsc+8nKBh54egMPLN7Et66ZjuN4\nevza/alN3VkiG2uG7Kw7G2uG3qs7kyBv7z0lYWVlfY+fGw4XUFHRdvGIvNnzCP36fzn0xLM0zT67\nw3NmTS7ntQ17WbVxH/cvepNLzhrb49fvL53VPdBlY82QnXVnY81w9HV3F/qZDK3sItUDbzYMeN9c\nAx/90HkABB/7U6f7HY+Hay+ZxKDCII8vfYcV6/Ycy+aJiPS5TIJ8MTAfwBhzKrDLWvu+eftsOmsm\n8TFjyXn8UTyHKjs9pjAU4D+vOJncoI97ntrAxm2dHyci4kZHDHJr7TJglTFmGakZK9cZYxYYYy4D\nMMb8CXg49aV5wRjziT5tcXuOQ8Onr8ETiZDz8INdHjYinM8XL58MwM8fe5OdFVqAQkQGBs+xvnd3\nRUVNj1+wqzElz8EDlJ58AvERI6lctgo8XQ/jL1+3h98+sZ5gwMvHzj6O2acMw9PN8e8H2TiGmI01\nQ3bWnY01Q4/GyLsMKtde2dlaclApjR++DN+WzfiXvtjtsTNOLOezl0zC6/HwwLOW2x55g/1VDceo\npSIivW9ABDlAw2euBSD3/nuOeOz0E8v57rVnMGV8Keu3VvLte19h+do9WllIRFxpwAR57LRpNJ00\nhcAzT+Ls3nXE40sKglw/fwqfufAEEkn47ZPr+c2iddTpjoki4jIDJsjxeIgsuAZPPE7uL9vfRaCr\np3iYOWUYN119OscNL+KVDfv4n7tf5lWt+ykiLjJwghyIXHElsbHjyP3tr/G9+krGzxtcnMtXr5rK\n5bPGUdfQxC8XruXnf36Tg9WRPmytiEjvGFBBTm4utT/7BZ5EgoLrvwCRzIPY6zhcfOYYbr7mDE4Y\nVcwbm/fz9d+uYNFL79DYFO/DRouIvDcDK8iBphln0XDNZ/G9tYm8n/zwqJ9fPijEDR+fytUXTiQn\n4GPhS+/w9btWsHT1LiJR3dtcRN5/BsQ88g5qaxk050ycndupevgxmubM7dHrNTTGeHrFNp59ZQex\neIKAz+GUCWWcMWkIk8eV4vMem/fBbJxnm401Q3bWnY01Q+/OIx+YQQ74l75I0fwP40kmic6ZS91X\nvkbs9DN69LoHqiIsXbOLlzfsY+/B1E2/CkJ+zpg0hDMmDWFEWT7BQM/v6Hgk2fiLno01Q3bWnY01\ng4I84+N9K5aTd+sPCKQvEmr45GeovfV2cHrWk04mk2zbW8PytXtZvm4PtQ2HpyoW5wcYVpbHaceH\nmTZxCPm5/h69Rmey8Rc9G2uG7Kw7G2sGBflRP8+3Yjn5X78B/9o1NCy4htof/bTby/gzEYsneHPL\nAd585yB7D9azr7KeA9WNAHgdDyeOHcTkcaVMHF3C0NLQe7oNQDb+omdjzZCddWdjzdC7Qd6T+5G7\nTmz6DKr+vIjiyy8h9/57SAaD1N18y3sKc5/XYerxYaYeH27ZVlnTyMvr97Ji/R7WbDnAmi0HACjK\nCzBlfClTJ4SZNKaEgL/vhmFEJPtkRY+8mWf/foovuxCf3UjTtDOIjz+O+LDhxMeMJT5xErEJBkKh\nrk+QTBL66Y8JLHmO6l/fQ2LEyC4P3XeogY3bKtm4rZL1Ww9SXZ8ahvH7HIryAuQGfeQGfZQWBhlc\nEmJwSS4ThhdRVpzb63W7UTbWDNlZdzbWDOqR91iyrIyqRxdR+Kkr8a98Gf/Kl9vu93iITzyRxnPP\nJ/qh84id+gHwHu49h370ffJ++mMAiuZ/mEOLniU5eHCnrzW4OJfBxbnMOnkYiWSSt3dV8/qmCtZt\nPUhNfRP7qxqINMbZ1O55I8L5nDKhlGGleQT8XgI+h7GxJAES+N/DEnkiMnBlVY+8jYYGvLvfxXn3\nXbyb38JnN+DdsB7/a6/iaUyNdcfLhxK56lNEPrmAnIcfJO+H3yM+egzReR8i997fEpt0EocWPkWy\nuKT710omOx3GiScSHKxuZF9lA7sO1LHunYOs31pJLJ7ocKzHA+GiXIaH8xg9pIDR5QUML8ujpDCI\nt4cf3r5nySSBJ/9CYsxYYpNP7tVTq5fWSiKR+gV4n99uuaf0s874+Oz+sPOo1NUR+McLBBY/Q3DR\nQpyaapKOgyeRID5yFIf+8gyJ4SPI/+qXyb3/HppOPY2an/2S+AkTD58jmcS3dg2BxX8lsPgZfOvW\nEvnYJ6j71s0ki4q7fflINMbGbYc4VNdItClBY1OchqYEb+88RHzzZuLVNWwtG9Pyj9rjSd0ArCgv\nSCgnNVwTCvrIy/WRn+unMBRgaGkeQ0tD5AY7/gEWiyeIROPk5fiO7gPZeJz8r99A7n13kygupnLJ\nMhLDR2T+/CM40s/a2bkD/8vLiZ5z7hG/p27Soe5olKL5H8apqqLqj4+TGFLe9ZOPJJHA2bmDxKjR\n772hRysWw1NdRXJQx3XaFeQZH68g75G6OnIef5Sc392Lp7aGqof+TGJMevHmRIKCf/8cOX96GIDG\niz5M5KMfx7/8nwSffgLv9m0AJH0+EmVhvHt2Ex88hNpbfkL04g+36V353lxN8I8P4+zehaeuFk99\nPbHJU4gsuJb4cRMI+2I0fOWr5Nx/D55EgoYhw3j7A3N49aRZrB58PAdqGqmqjRJPHP7WltQe5LJV\nf2HM/q0sm3AmL5wwi0BxIQG/F5/XweOBmvqmlimUJQVBjh9ZzPhhhS2B7/FAKOgnP9dPfshPcX6A\nnIAPIhEKP38twacWER88BO++vURnnEXVY0+2GYpq5n1rE57KyqOax9/Zz9pTW0PuL+4k+MxT+Nav\nBaDp5KlU/XkRycKijq+7fh2hO28jNvU0Gj77hbY92vp6PI0RkiWDMm7TsdC+7rzv30TojtsAiJkT\nOPT40yTLygDwvb4K37q1xCZOIjbxxG4/3/G/uIS8734b/5o3qP2f79DwH1/u9bZ77UZCP/sJxGLU\n/ui2ltD2VFRQ9In5+DZt5NDCp4lNPa3N89wY5P7l/yTvG18ldspUoudfSHTmHMjt+PlWdxTk7xfJ\nJIHFfyX00x/hf/21ls2J/AKi555H9PyLiJ49j2Qoj9Av7iB024/wNDaSKAvTNO0MYpNOJPDcYvxv\nvN7lS0RnziawcT1UVBAbfxyxk08h8Pe/4VRXARAfM5bIlVfROHMO8foGmqqqCSx5jkF/ehAn2thy\nnkgwxOrjTycYqaekuoLcSD17ysewZ+xEKkZOoHrPfkKVFRQ01GKHHs+qsafS6M9p05acaAOztr3K\nJa8/yZidlrfGn8xdn7qZT/zxVqaue4lHz/4Uz8y9isK8AIWhAKV5PmYveYTJD/wcJ9bE3g9/lI3/\n8Q1ieYUU5qX+WsjP9ePzOTgeD8lkkmgsQX0kRn5BDrU1EXICXoIBL979+yn6+Efwr3mDZCBA5MyZ\nRHEoeuFvRKefSdXDj7UEmbN7F6EffZ+chx/Ek0gNU9V/7ovU3fR98HgIPLmIgi99EU9tDdE5c2m8\n4koaz7+o2yB03t2Jd5Ol6ayZEAhk9Ovhqa3BU1GBc6gST10dTaecCvn53T6n9e+4b8Vyiv/fBSRG\njiI69xxy77ubppOmUPfdW8j95Z0E//Zsy/OSjkPcnEDTadOInTaN2PgJOPv24N2xg8CS5wj8YwkA\niYJCnJpqqn73ENELLsqojiPxvrWJ0E9uIbjwMTzpPImPHEX1fX8gUVhE0ccuw/fO2y3bK//+jzZv\noJ3+u47F8K16ldjUUzP6fns3WQqu/zyxE6dQe/MPup+08B457+6k5EOzcPbvb9mWDOVR9/Vv0vCv\nn894COyYB7kx5nZgOpAErrfWrmy17xzgB0AceNpa+93uzjWggrxZMon/hecJLH2RphlnEp11NgSD\nHQ7zbnmL0K0/xL/8n3jT90xPOg7RD51H5JMLaDrlNJJ5eeD3E3zmSXLuuYvAimUQClH75f+m4d+u\nS503GsX/z6XkPPoIwSf/gqeh4wpH8VGjqf+PLxOdM5ecPz5Ezu/vP/yaoTySeXk4Ffu6LCkeCLL3\ntLOozysk0RDBW1PNyA2rCDSl3hz+MXEWv7jgemJ+P0WNddx6779TXHuQX136FXYGivDG43x8+SOc\n9O46DuaVcChUxLiKrVTkl/L7D/4LB/JLiaTfKIoaqihpqGZQ7UHKK3cx9NAe8hrreHn86fztpHkk\n8fDdx29iaOVuXpp2Ab+dfTUHE36cRJyvPP1TZm76J5smns6OU87k+NUvMXzj6ziJOAdGjGfFxZ9m\n+qL7KN31DutnXUI8FGLyXx8hFsyhesQ4Bm1ZD0BdfjHPz72S56ZdTCInlyGDQpQPCjHcG2XSI3cx\n4k8P4EQbaRxczrYrr2brhR/FX1JETsBHwOcQiyeJNsVpijZRtuIFhj/2fwxa/kLLGwlArKSUiqs/\nz+4rPo23MD89c8nbcqsHjwfC4UL276/Bqalm0NwP4uzcwaFFzxKbdjr5N3yJ3AfubTlfdPqZ1F96\nOYn1G/GvXUNow5t4I52vdhWdM5e6b94EQPEl5wEeKp/6G/ETT2o5xnOoEv/yZfhXvkwyECBRPpRE\n+VBip0wlUT60wzk9lQcJ3XoLuffdjScep2nyydTf8DV8b64m79ZbSObkkMzPx9m/n7ovfQUcL3m3\n/YjGc8+n+oGHIRol9/67yV/zGlWXfZToOeeBx4OzbSuFX/hX/CtfJjZ2HHXfvJnoRZd0GZCBZ5+h\n4PPX4tSmsiE26SSq732A+Ljjuvz9zpR3y1t4162lae45JPMLIBKh+NLz8b/+GjU/+DGxKVMJPvs0\nOQ/9Hmf/fiKXz6fmtp9DXt7h79O+fQReehH/ypeJXPoRYtNnAMc4yI0xs4EbrLUXG2MmAvdaa2e0\n2r8eOA94F3gR+Ddr7fquzjcgg7wHnHd34ntzDbHJU7odW3beeZvSccOpSHZ8YwDw1FQTfOIveO1G\nknl5JPPyiY8cSfSCi8Hf6urDiu0sAAAKx0lEQVTSWAzv9q0kysIkCwrB48Gzbx/+N1bh3bSJZEkJ\niSFDSIby8L/4PMGnnsC3ybZ5rdi48TRefgWNl19B/LgJbfb5l71E0WUXtfTImu2aeS5/+8yNHHJy\nmfH07/jAn+/GiXd/87GExyHh8+NLv2k0BnIIRiP8ZeaVPHjWVRQXBAkX51KcH6TqYA3zf3kjp7zV\n0rfAlk/g2cnn8tyJc0k4XgobqvnOYzczYe9mALaVjuLHF32F7WWjGHFwJ2evX8KFq58hv7GeyrwS\nXjrhg/ijEfIjdZy8fQ0FjbVUFJTx2uipzLJLyW2K0OgLUFEQ5kD+IGpyCshpihCK1jOkai+ldZUA\nvDVkPNvKRlOTU4CTTDBv3XOp1wgVsXLcNLaWjWZb2SjyGusZdWA7Iw/sJLepgbjHS1ntfibs3cIj\nZ1zBH866Csfjwe8kuXrJPYyq2MZjZ8zntZFTiLf6djuJOKP3b2PSnk2MbjhATUmYmrJyDgwewdZB\no2hojNHYFOfMzcv54sPfo7JkCJsmTqOw6gDFlfsYsvttnC7yYNvQ43jjuGkcKiwjPx6hKFLDjFee\nJq++mj2DhvHwOVezyszA4zjkBLycuf01rvz99wg21LHkk//FK3Pmk4zHueqOLzFq7StsmXUh4XWr\nKDywt+U13h5/Mm9NOYs5z9xPMFLPngmTCb+9Hm88zttjJ7Nmykz2Dh3H/vLRDPJEGVZ3gHGbX+ek\nx+8n5g/wx498iZHvrGfWikVEgiGen/txdo2bxL4xE2FQCaEcP3k5qTdef6QBf6SOZFOMem+Aenw0\nev14vV68XofwOxuYuvBejnvleTzJJJH8QjZfchWFB3YzavFCtsy9lBe+eBOQytZQ5T7m/vi/GGLX\ncHDEOPaMP5G8gxUU7N/NoN3bWmrc/cUb8H3rm8CxD/Kbge3W2rvTjzcCp1trq40x44AHrLUfTO/7\nGlBrrf15V+dTkB+9/qrb2bkDmpogJ4dkMJj6c7ibPxv9LzyPf/lLqb/bgNjkKUQvvrTNc7wb1hN4\n/u946utSf0nE4yRKy0iEwyQHDyY+dhzxEaMIl+RSfd8fyHnoD/hfWUHdd76X+rO1Mw0N+H/6E+rC\n5eyfcTZVhaV4HQ8Bv4Pf58XxQLK6hhE3fZXGomLsZ2+gzuPH63goyg9QlBegsLGekvt+ReiuX+HU\n1bacujG/kDXzr+XVufNpcPwUR+uY+rc/MmbFc+Qe2EdOdWXLsQnHobGgiG3Tz2HjuR9h3+gTaGyK\nE4nGiETjhCJ1nPncI5z67EMEG+qO+P3fMv5kfvO5nxD3+UkkksTiCWLxJEmS+LwOPq+HgM+ban9e\nAK/j4UB1hIpDDRysSX1YHm2KE08kyQl4yQ36CPq9NDbFOf/5B/nE0j8crtMX4K0hE1g39mQ2j51M\nIgmF1QcYdGgfU7av4cTta/El2r4B1wVCPD7z4yyZcWnqL8UkJElSH4lRF4kxqOYAJfWH2DJkfMtz\niuoPccfvv0xp3UEavQGeOuVCXj1hBpcuf5Qz3k69GdcHcvn13M+yZOIchlfuYsHS3zF9S9frC+wr\nCPP9S7/GO0PG4fc5zFz7Ap9b/Atymg4PLTb6Ai2/l75EDG+y48ywlp8jHpz0wZsHj2P1qCmcs+55\nihqqW7Z99WO3EPW37Vz54k1c/eJ9XPLG0y3baoL5bC4fzxujTmb1qClMuPhsPjov1Qk61kF+F/CU\ntfYv6cdLgWustZuMMWeS6q1flt53DTDeWvv1rs4Xi8WTPs2HlqMRi4HvGF3ycPAgbNoExcUwaFDq\nv+5eu7ERKitTY995eZmNj0ajqdd4801Ytw4KC+HEE1P/lZRAPJ76r6ysV6YcJpPJTmckxddvIJlI\nkCwvJ1lYhN/v7XrmUk0NLFkCtbWp9hYUwOTJqe9PJyKNMSoONVAfaSIn4CMYSJ27rqGJptdep+CZ\nJ0hccy2DTjyOUI6fZDJJ/MV/kHjscao/81nqykcQicbID6XeaPN2vINn9WqSa9eStJZoKJ/a8FCq\nSstp+tD5DBo7jNLCHLzNdyTdu5fk0qUkVq0iuep1kgf2k0gkiScg7vMTz88nnpcPXh++pgi+xghO\nNAqJBMlEgnhRMdWfvpams+eRBGorKgk9+AD5ryzj7a98G2fMmJaamt/AvI6D1+shZ/s7OI4XhpXj\nCeWRSCZTExGSMGZY4Xu5a2qvBvlLwNVdBPm1wLjuglw98qOXjXVnY82QnXVnY83Quz3yTN4adgGt\nJ68OA3Z3sW94epuIiBwjmQT5YmA+gDHmVGCXtbYGwFq7FSg0xowxxviAi9PHi4jIMXLEgUdr7TJj\nzCpjzDIgAVxnjFkAVFlrHwc+DzyUPvwRa23724eIiEgfyugTJGvtje02rW617x/ADEREpF8MuMWX\nRUSyjYJcRMTlFOQiIi6nIBcRcbljfvdDERHpXeqRi4i4nIJcRMTlFOQiIi6nIBcRcTkFuYiIyynI\nRURcTkEuIuJyx2jZlfeuuwWgBxpjzI+BmaR+PrcAK4HfA15S94L/pLW2seszuJMxJhdYC3wXeI7s\nqPkq4L+BGPAtYA0DuG5jTD7wAFACBIGbgD3Ar0j9215jre1iTT/3McacBPwFuN1a+7/GmJF08vNN\n/x78J6k7zN5lrb3naF7HFT3y9ALQE9KLPl8D3NnPTeozxpizgZPStZ4P/Ay4GfiFtXYmsBm4uh+b\n2Jf+BziY/nrA12yMKQW+DXyQ1L38L2Xg170AsNbas0mtc3AHqd/x6621ZwFFxpgL+rF9vcYYkwf8\nnFSnpFmHn2/6uG8B5wBzgC8ZYzpfQ68LrghyYB6wEMBauwEoMcYU9m+T+sw/gCvSXx8C8kj9cBel\ntz1B6gc+oBhjTgAmAU+lN81hgNdMqqa/W2trrLW7rbWfZeDXvR8oTX9dQuqNe2yrv7AHUs2NwIW0\nXTVtDh1/vmcAK621VdbaBuCfwFlH80JuCfJyoKLV4wraLjE3YFhr49ba5iXWrwGeBvJa/Xm9Dxja\nL43rW7cBX271OBtqHgOEjDGLjDFLjTHzGOB1W2sfBkYZYzaT6rR8BahsdciAqdlaG0sHc2ud/Xzb\n59tRfw/cEuTtvfelxd/njDGXkgryL7bbNeBqN8Z8ClhurX2ni0MGXM1pHlK908tJDTncR9taB1zd\nxph/AbZba48D5gJ/aHfIgKu5G13VetTfA7cEeXcLQA84xpjzgG8AF1hrq4Da9AeBMDAXuL4IuNQY\nswK4FvgmA79mgL3AsnTPbQtQA9QM8LrPAp4FsNauBnKBslb7B2LNrXX2e/2eF7F3S5B3uQD0QGOM\nKQJuBS621jZ/8Pd34CPprz8C/LU/2tZXrLUfs9ZOs9ZOB+4mNWtlQNecthiYa4xx0h985jPw695M\nakwYY8xoUm9eG4wxH0zvv5yBV3Nrnf18XwamGWOK07N6zgKWHs1JXXMbW2PMD4FZpBeATr+bDzjG\nmM8C3wFaL2L9aVIBlwNsAz5jrW069q3re8aY7wBbSfXaHmCA12yM+TdSQ2gA3yM11XTA1p0OqnuB\nIaSm136T1PTD35DqWL5srf1y12dwD2PMaaQ++xkDNAHvAlcB99Pu52uMmQ/cQGoK5s+ttQ8ezWu5\nJshFRKRzbhlaERGRLijIRURcTkEuIuJyCnIREZdTkIuIuJyCXETE5RTkIiIu9/8Bqjv/5VEvUQEA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(H.history['loss'])\n",
    "plt.plot(H.history['val_loss'],'r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4RIGVSojmy-R"
   },
   "source": [
    "### Predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tnigE74rm39a"
   },
   "outputs": [],
   "source": [
    "unlabeled_images_test = pd.read_csv('gdrive/My Drive/dataML/test.csv')\n",
    "#unlabeled_images_test = pd.read_csv('test.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3Fotf1KrpXVE"
   },
   "outputs": [],
   "source": [
    "X_unlabeled = unlabeled_images_test.values.reshape(unlabeled_images_test.shape[0],28,28,1)/255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jeRnCHzdutQ0"
   },
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_unlabeled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3M_fePteu-3X"
   },
   "outputs": [],
   "source": [
    "y_label = np.argmax(y_pred, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RX5PuSUmvRri"
   },
   "source": [
    "### Save csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zU5Q1fSRvVbn"
   },
   "outputs": [],
   "source": [
    "imageId = np.arange(1,y_label.shape[0]+1).tolist()\n",
    "prediction_pd = pd.DataFrame({'ImageId':imageId, 'Label':y_label})\n",
    "prediction_pd.to_csv('gdrive/My Drive/dataML/out_cnn08.csv',sep = ',', index = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_qetEX7AgBQY"
   },
   "source": [
    "# Tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4keUL7d0gBQZ"
   },
   "source": [
    "### Helper functions for batch learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "czdbjPfcgBQd"
   },
   "outputs": [],
   "source": [
    "def one_hot_encode(vec, vals=10):\n",
    "    '''\n",
    "    For use to one-hot encode the 10- possible labels\n",
    "    '''\n",
    "    n = len(vec)\n",
    "    out = np.zeros((n, vals))\n",
    "    out[range(n), vec] = 1\n",
    "    return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Afc2XmK2gBQh"
   },
   "outputs": [],
   "source": [
    "class CifarHelper():\n",
    "    \n",
    "    def __init__(self):\n",
    "        self.i = 0\n",
    "        \n",
    "        # Intialize some empty variables for later on\n",
    "        self.training_images = None\n",
    "        self.training_labels = None\n",
    "        \n",
    "        self.test_images = None\n",
    "        self.test_labels = None\n",
    "    \n",
    "    def set_up_images(self):\n",
    "        \n",
    "        print(\"Setting Up Training Images and Labels\")\n",
    "        \n",
    "        # Vertically stacks the training images\n",
    "        self.training_images = train_images.as_matrix()\n",
    "        train_len = self.training_images.shape[0]\n",
    "        \n",
    "        # Reshapes and normalizes training images\n",
    "        self.training_images = self.training_images.reshape(train_len,28,28,1)/255\n",
    "        # One hot Encodes the training labels (e.g. [0,0,0,1,0,0,0,0,0,0])\n",
    "        self.training_labels = one_hot_encode(train_labels.as_matrix().reshape(-1), 10)\n",
    "        \n",
    "        print(\"Setting Up Test Images and Labels\")\n",
    "        \n",
    "        # Vertically stacks the test images\n",
    "        self.test_images = test_images.as_matrix()\n",
    "        test_len = self.test_images.shape[0]\n",
    "        \n",
    "        # Reshapes and normalizes test images\n",
    "        self.test_images = self.test_images.reshape(test_len,28,28,1)/255\n",
    "        # One hot Encodes the test labels (e.g. [0,0,0,1,0,0,0,0,0,0])\n",
    "        self.test_labels = one_hot_encode(test_labels.as_matrix().reshape(-1), 10)\n",
    "\n",
    "        \n",
    "    def next_batch(self, batch_size):\n",
    "        # Note that the 100 dimension in the reshape call is set by an assumed batch size of 100\n",
    "        x = self.training_images[self.i:self.i+batch_size]\n",
    "        y = self.training_labels[self.i:self.i+batch_size]\n",
    "        self.i = (self.i + batch_size) % len(self.training_images)\n",
    "        return x, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "1qAiPT1-gBQp",
    "outputId": "579a01c2-db46-4e51-8279-eb66d5844e35"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting Up Training Images and Labels\n",
      "Setting Up Test Images and Labels\n"
     ]
    }
   ],
   "source": [
    "# Before Your tf.Session run these two lines\n",
    "ch = CifarHelper()\n",
    "ch.set_up_images()\n",
    "\n",
    "# During your session to grab the next batch use this line\n",
    "# (Just like we did for mnist.train.next_batch)\n",
    "# batch = ch.next_batch(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QUkUv8sKgBQw"
   },
   "source": [
    "## Creating the Model\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l1XSMzIpgBQx"
   },
   "source": [
    "** Create 2 placeholders, x and y_true. Their shapes should be: **\n",
    "\n",
    "* X shape = [None,28,28,1]\n",
    "* Y_true shape = [None,10]\n",
    "\n",
    "** Create three more placeholders \n",
    "* lr: learning rate\n",
    "* step：for learning rate decay\n",
    "* drop_rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8Y_4DDQvgBQz"
   },
   "outputs": [],
   "source": [
    "X = tf.placeholder(tf.float32, shape=[None,28,28,1])\n",
    "Y_true = tf.placeholder(tf.float32, shape=[None,10])\n",
    "\n",
    "lr = tf.placeholder(tf.float32)\n",
    "step = tf.placeholder(tf.int32)\n",
    "drop_rate = tf.placeholder(tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5egQVsS4NhxW"
   },
   "source": [
    "### Initialize Weights and bias\n",
    " neural network structure for this sample:\n",
    "\n",
    "X [batch, 28, 28, 1]\n",
    " \n",
    "Layer 1:  conv. layer 6x6x1=>6, stride 1    W1 [6, 6, 1, 6] ,       B1 [6]\n",
    "\n",
    "Y1 [batch, 28, 28, 6]\n",
    "\n",
    "Layer 2: conv. layer 5x5x6=>12, stride 2       W2 [5, 5, 6, 12] ,       B2 [12]\n",
    "\n",
    "Y2 [batch, 14, 14, 12]\n",
    "\n",
    "Layer 3: conv. layer 4x4x12=>24, stride 2      W3 [4, 4, 12, 24]  ,     B3 [24]\n",
    "\n",
    "Y3 [batch, 7, 7, 24] => reshaped to YY [batch, 7*7*24]\n",
    "\n",
    "Layer 4: fully connected layer (relu+dropout), W4 [7*7*24, 200]       B4 [200]\n",
    "\n",
    "Y4 [batch, 200]\n",
    "\n",
    "Layer 5: fully connected layer (softmax)      W5 [200, 10]           B5 [10]\n",
    "\n",
    "Y [batch, 10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UD6cWs60OnGn"
   },
   "outputs": [],
   "source": [
    "# three convolutional layers with their channel counts, and a\n",
    "# fully connected layer (the last layer has 10 softmax neurons)\n",
    "K = 12  # first convolutional layer output depth\n",
    "L = 24  # second convolutional layer output depth\n",
    "M = 48  # third convolutional layer\n",
    "N = 200  # fully connected layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xiVQdlDsN1W-"
   },
   "outputs": [],
   "source": [
    "W1 = tf.Variable(tf.truncated_normal([6,6,1,K], stddev=0.1)) \n",
    "B1 = tf.Variable(tf.ones([K])/10)\n",
    "\n",
    "W2 = tf.Variable(tf.truncated_normal([5,5,K,L], stddev=0.1))\n",
    "B2 = tf.Variable(tf.ones([L])/10)\n",
    "\n",
    "W3 = tf.Variable(tf.truncated_normal([4,4,L,M], stddev=0.1))\n",
    "B3 = tf.Variable(tf.ones([M])/10)\n",
    "\n",
    "W4 = tf.Variable(tf.truncated_normal([7*7*M,N], stddev=0.1))\n",
    "B4 = tf.Variable(tf.ones([N])/10)\n",
    "\n",
    "W5 = tf.Variable(tf.truncated_normal([N, 10], stddev=0.1))\n",
    "B5 = tf.Variable(tf.zeros([10]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "u4RDJq8pO_Og"
   },
   "source": [
    "### layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DM5_O098O4Di"
   },
   "outputs": [],
   "source": [
    "Y1 = tf.nn.relu(tf.nn.conv2d(X, W1, strides = [1,1,1,1], padding='SAME') + B1)\n",
    "\n",
    "Y2 = tf.nn.relu(tf.nn.conv2d(Y1,W2, strides = [1,2,2,1], padding='SAME') + B2)\n",
    "\n",
    "Y3 = tf.nn.relu(tf.nn.conv2d(Y2,W3, strides = [1,2,2,1], padding='SAME') + B3)\n",
    "\n",
    "#flat the inputs for the fully connected nn\n",
    "YY3 = tf.reshape(Y3, shape = (-1,7*7*M))\n",
    "                \n",
    "\n",
    "Y4 = tf.nn.relu(tf.matmul(YY3, W4) + B4)\n",
    "Y4d = tf.nn.dropout(Y4,rate = drop_rate)\n",
    "\n",
    "Ylogits = tf.matmul(Y4d, W5) + B5\n",
    "Y = tf.nn.softmax(Ylogits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tT4TvNz-gBRI"
   },
   "source": [
    "### Loss Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 88
    },
    "colab_type": "code",
    "id": "wkqQ5GurgBRJ",
    "outputId": "9145b309-a37b-4f3a-9b69-0b99eec766fc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n"
     ]
    }
   ],
   "source": [
    "#cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=Y_true,logits=Ylogits))\n",
    "cross_entropy = tf.losses.softmax_cross_entropy(onehot_labels = Y_true, logits = Ylogits)\n",
    "#cross_entropy = -tf.reduce_mean(y_true * tf.log(Ylogits)) * 1000.0 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jmnEUVWxgBRM"
   },
   "source": [
    "### Optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MoyIlzCagBRN"
   },
   "outputs": [],
   "source": [
    "lr = 0.0001 + tf.train.exponential_decay(learning_rate = 0.003, \n",
    "                                         global_step = step,\n",
    "                                         decay_steps = 2000,\n",
    "                                         decay_rate = 1/math.e\n",
    "                                        )\n",
    "\n",
    "#optimizer = tf.train.GradientDescentOptimizer(learning_rate = 0.005)\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate=lr)\n",
    "train = optimizer.minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "47rAzeVNgBRP"
   },
   "source": [
    "### Intialize Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7WG1AszIgBRQ"
   },
   "outputs": [],
   "source": [
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rseSYLjggBRb"
   },
   "source": [
    "### Saving the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OwdUgOG4gBRc"
   },
   "outputs": [],
   "source": [
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xGdHTpE0gBRe"
   },
   "source": [
    "## Graph Session\n",
    "\n",
    "** Perform the training and test print outs in a Tf session and run your model! **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 10237
    },
    "colab_type": "code",
    "id": "pQHEYbyZgBRf",
    "outputId": "cc718ec8-8ef0-42c4-b230-898ac94d6e09"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 0:\tlearning_rate=0.003100,\tloss_train=3.291209,\tloss_val=3.566376,\tacc_train=0.150000,\tacc_val=0.080952\n",
      "\n",
      "\n",
      "Iteration 100:\tlearning_rate=0.002954,\tloss_train=0.273934,\tloss_val=0.113660,\tacc_train=0.950000,\tacc_val=0.973810\n",
      "\n",
      "\n",
      "Iteration 200:\tlearning_rate=0.002815,\tloss_train=0.131737,\tloss_val=0.094597,\tacc_train=0.980000,\tacc_val=0.969048\n",
      "\n",
      "\n",
      "Iteration 300:\tlearning_rate=0.002682,\tloss_train=0.058850,\tloss_val=0.089490,\tacc_train=0.980000,\tacc_val=0.971429\n",
      "\n",
      "\n",
      "Iteration 400:\tlearning_rate=0.002556,\tloss_train=0.052930,\tloss_val=0.051649,\tacc_train=0.990000,\tacc_val=0.983333\n",
      "\n",
      "\n",
      "Iteration 500:\tlearning_rate=0.002436,\tloss_train=0.108525,\tloss_val=0.043813,\tacc_train=0.970000,\tacc_val=0.980952\n",
      "\n",
      "\n",
      "Iteration 600:\tlearning_rate=0.002322,\tloss_train=0.077652,\tloss_val=0.043063,\tacc_train=0.980000,\tacc_val=0.983333\n",
      "\n",
      "\n",
      "Iteration 700:\tlearning_rate=0.002214,\tloss_train=0.062273,\tloss_val=0.050851,\tacc_train=0.990000,\tacc_val=0.983333\n",
      "\n",
      "\n",
      "Iteration 800:\tlearning_rate=0.002111,\tloss_train=0.055581,\tloss_val=0.016286,\tacc_train=1.000000,\tacc_val=0.995238\n",
      "\n",
      "\n",
      "Iteration 900:\tlearning_rate=0.002013,\tloss_train=0.063747,\tloss_val=0.026281,\tacc_train=0.990000,\tacc_val=0.985714\n",
      "\n",
      "\n",
      "Iteration 1000:\tlearning_rate=0.001920,\tloss_train=0.037742,\tloss_val=0.045727,\tacc_train=0.990000,\tacc_val=0.985714\n",
      "\n",
      "\n",
      "Iteration 1100:\tlearning_rate=0.001831,\tloss_train=0.020697,\tloss_val=0.022253,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 1200:\tlearning_rate=0.001746,\tloss_train=0.025594,\tloss_val=0.020167,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 1300:\tlearning_rate=0.001666,\tloss_train=0.005705,\tloss_val=0.022109,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 1400:\tlearning_rate=0.001590,\tloss_train=0.015113,\tloss_val=0.027408,\tacc_train=0.990000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 1500:\tlearning_rate=0.001517,\tloss_train=0.072400,\tloss_val=0.028506,\tacc_train=0.980000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 1600:\tlearning_rate=0.001448,\tloss_train=0.023121,\tloss_val=0.055556,\tacc_train=1.000000,\tacc_val=0.983333\n",
      "\n",
      "\n",
      "Iteration 1700:\tlearning_rate=0.001382,\tloss_train=0.014415,\tloss_val=0.019530,\tacc_train=1.000000,\tacc_val=0.995238\n",
      "\n",
      "\n",
      "Iteration 1800:\tlearning_rate=0.001320,\tloss_train=0.003565,\tloss_val=0.022693,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 1900:\tlearning_rate=0.001260,\tloss_train=0.015690,\tloss_val=0.055221,\tacc_train=1.000000,\tacc_val=0.983333\n",
      "\n",
      "\n",
      "Iteration 2000:\tlearning_rate=0.001204,\tloss_train=0.001746,\tloss_val=0.033166,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 2100:\tlearning_rate=0.001150,\tloss_train=0.005184,\tloss_val=0.042062,\tacc_train=0.980000,\tacc_val=0.985714\n",
      "\n",
      "\n",
      "Iteration 2200:\tlearning_rate=0.001099,\tloss_train=0.019183,\tloss_val=0.046565,\tacc_train=0.990000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 2300:\tlearning_rate=0.001050,\tloss_train=0.000312,\tloss_val=0.018573,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 2400:\tlearning_rate=0.001004,\tloss_train=0.002007,\tloss_val=0.040470,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 2500:\tlearning_rate=0.000960,\tloss_train=0.003134,\tloss_val=0.020767,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 2600:\tlearning_rate=0.000918,\tloss_train=0.002078,\tloss_val=0.027919,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 2700:\tlearning_rate=0.000878,\tloss_train=0.007485,\tloss_val=0.020951,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 2800:\tlearning_rate=0.000840,\tloss_train=0.003849,\tloss_val=0.022948,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 2900:\tlearning_rate=0.000804,\tloss_train=0.005030,\tloss_val=0.014846,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 3000:\tlearning_rate=0.000769,\tloss_train=0.000572,\tloss_val=0.021942,\tacc_train=1.000000,\tacc_val=0.995238\n",
      "\n",
      "\n",
      "Iteration 3100:\tlearning_rate=0.000737,\tloss_train=0.000241,\tloss_val=0.014402,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 3200:\tlearning_rate=0.000706,\tloss_train=0.001775,\tloss_val=0.033364,\tacc_train=0.990000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 3300:\tlearning_rate=0.000676,\tloss_train=0.000224,\tloss_val=0.018216,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 3400:\tlearning_rate=0.000648,\tloss_train=0.002972,\tloss_val=0.012856,\tacc_train=1.000000,\tacc_val=0.995238\n",
      "\n",
      "\n",
      "Iteration 3500:\tlearning_rate=0.000621,\tloss_train=0.000103,\tloss_val=0.039060,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 3600:\tlearning_rate=0.000596,\tloss_train=0.050762,\tloss_val=0.057317,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 3700:\tlearning_rate=0.000572,\tloss_train=0.000142,\tloss_val=0.038090,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 3800:\tlearning_rate=0.000549,\tloss_train=0.000966,\tloss_val=0.033859,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 3900:\tlearning_rate=0.000527,\tloss_train=0.002678,\tloss_val=0.037520,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 4000:\tlearning_rate=0.000506,\tloss_train=0.000048,\tloss_val=0.043038,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 4100:\tlearning_rate=0.000486,\tloss_train=0.000235,\tloss_val=0.035163,\tacc_train=1.000000,\tacc_val=0.983333\n",
      "\n",
      "\n",
      "Iteration 4200:\tlearning_rate=0.000467,\tloss_train=0.001476,\tloss_val=0.041278,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 4300:\tlearning_rate=0.000449,\tloss_train=0.001006,\tloss_val=0.046137,\tacc_train=1.000000,\tacc_val=0.983333\n",
      "\n",
      "\n",
      "Iteration 4400:\tlearning_rate=0.000432,\tloss_train=0.000506,\tloss_val=0.035295,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 4500:\tlearning_rate=0.000416,\tloss_train=0.000803,\tloss_val=0.031128,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 4600:\tlearning_rate=0.000401,\tloss_train=0.002269,\tloss_val=0.037874,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 4700:\tlearning_rate=0.000386,\tloss_train=0.000586,\tloss_val=0.035575,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 4800:\tlearning_rate=0.000372,\tloss_train=0.000072,\tloss_val=0.032569,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 4900:\tlearning_rate=0.000359,\tloss_train=0.004966,\tloss_val=0.041759,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 5000:\tlearning_rate=0.000346,\tloss_train=0.000024,\tloss_val=0.023334,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 5100:\tlearning_rate=0.000334,\tloss_train=0.000289,\tloss_val=0.027374,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 5200:\tlearning_rate=0.000323,\tloss_train=0.000012,\tloss_val=0.038204,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 5300:\tlearning_rate=0.000312,\tloss_train=0.003487,\tloss_val=0.037891,\tacc_train=1.000000,\tacc_val=0.985714\n",
      "\n",
      "\n",
      "Iteration 5400:\tlearning_rate=0.000302,\tloss_train=0.000134,\tloss_val=0.031178,\tacc_train=1.000000,\tacc_val=0.985714\n",
      "\n",
      "\n",
      "Iteration 5500:\tlearning_rate=0.000292,\tloss_train=0.026467,\tloss_val=0.026724,\tacc_train=0.990000,\tacc_val=0.995238\n",
      "\n",
      "\n",
      "Iteration 5600:\tlearning_rate=0.000282,\tloss_train=0.000355,\tloss_val=0.025337,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 5700:\tlearning_rate=0.000274,\tloss_train=0.001310,\tloss_val=0.032937,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 5800:\tlearning_rate=0.000265,\tloss_train=0.000005,\tloss_val=0.033200,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 5900:\tlearning_rate=0.000257,\tloss_train=0.000055,\tloss_val=0.043227,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 6000:\tlearning_rate=0.000249,\tloss_train=0.002238,\tloss_val=0.050428,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 6100:\tlearning_rate=0.000242,\tloss_train=0.000573,\tloss_val=0.031664,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 6200:\tlearning_rate=0.000235,\tloss_train=0.000065,\tloss_val=0.036321,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 6300:\tlearning_rate=0.000229,\tloss_train=0.000200,\tloss_val=0.028852,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 6400:\tlearning_rate=0.000222,\tloss_train=0.000258,\tloss_val=0.030592,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 6500:\tlearning_rate=0.000216,\tloss_train=0.000016,\tloss_val=0.022441,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 6600:\tlearning_rate=0.000211,\tloss_train=0.007413,\tloss_val=0.032315,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 6700:\tlearning_rate=0.000205,\tloss_train=0.001362,\tloss_val=0.034481,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 6800:\tlearning_rate=0.000200,\tloss_train=0.000106,\tloss_val=0.037938,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 6900:\tlearning_rate=0.000195,\tloss_train=0.001134,\tloss_val=0.046206,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 7000:\tlearning_rate=0.000191,\tloss_train=0.000098,\tloss_val=0.040202,\tacc_train=0.990000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 7100:\tlearning_rate=0.000186,\tloss_train=0.000029,\tloss_val=0.037919,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 7200:\tlearning_rate=0.000182,\tloss_train=0.002708,\tloss_val=0.049570,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 7300:\tlearning_rate=0.000178,\tloss_train=0.000150,\tloss_val=0.065175,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 7400:\tlearning_rate=0.000174,\tloss_train=0.000034,\tloss_val=0.036139,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 7500:\tlearning_rate=0.000171,\tloss_train=0.000393,\tloss_val=0.045847,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 7600:\tlearning_rate=0.000167,\tloss_train=0.000328,\tloss_val=0.057202,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 7700:\tlearning_rate=0.000164,\tloss_train=0.000026,\tloss_val=0.045608,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 7800:\tlearning_rate=0.000161,\tloss_train=0.001196,\tloss_val=0.047827,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 7900:\tlearning_rate=0.000158,\tloss_train=0.000038,\tloss_val=0.043284,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 8000:\tlearning_rate=0.000155,\tloss_train=0.000109,\tloss_val=0.041985,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 8100:\tlearning_rate=0.000152,\tloss_train=0.000008,\tloss_val=0.057204,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 8200:\tlearning_rate=0.000150,\tloss_train=0.000007,\tloss_val=0.061682,\tacc_train=1.000000,\tacc_val=0.985714\n",
      "\n",
      "\n",
      "Iteration 8300:\tlearning_rate=0.000147,\tloss_train=0.000008,\tloss_val=0.056300,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 8400:\tlearning_rate=0.000145,\tloss_train=0.000237,\tloss_val=0.053918,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 8500:\tlearning_rate=0.000143,\tloss_train=0.000021,\tloss_val=0.060757,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 8600:\tlearning_rate=0.000141,\tloss_train=0.000096,\tloss_val=0.056191,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 8700:\tlearning_rate=0.000139,\tloss_train=0.000090,\tloss_val=0.043130,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 8800:\tlearning_rate=0.000137,\tloss_train=0.000013,\tloss_val=0.044164,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 8900:\tlearning_rate=0.000135,\tloss_train=0.001191,\tloss_val=0.056158,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 9000:\tlearning_rate=0.000133,\tloss_train=0.002281,\tloss_val=0.063807,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 9100:\tlearning_rate=0.000132,\tloss_train=0.000023,\tloss_val=0.064389,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 9200:\tlearning_rate=0.000130,\tloss_train=0.000010,\tloss_val=0.062237,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 9300:\tlearning_rate=0.000129,\tloss_train=0.000011,\tloss_val=0.055023,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 9400:\tlearning_rate=0.000127,\tloss_train=0.001363,\tloss_val=0.060808,\tacc_train=0.990000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 9500:\tlearning_rate=0.000126,\tloss_train=0.000015,\tloss_val=0.069666,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 9600:\tlearning_rate=0.000125,\tloss_train=0.000559,\tloss_val=0.066412,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 9700:\tlearning_rate=0.000123,\tloss_train=0.000048,\tloss_val=0.049534,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 9800:\tlearning_rate=0.000122,\tloss_train=0.000039,\tloss_val=0.056987,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 9900:\tlearning_rate=0.000121,\tloss_train=0.000081,\tloss_val=0.049058,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 10000:\tlearning_rate=0.000120,\tloss_train=0.000057,\tloss_val=0.044235,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 10100:\tlearning_rate=0.000119,\tloss_train=0.000252,\tloss_val=0.047751,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 10200:\tlearning_rate=0.000118,\tloss_train=0.000002,\tloss_val=0.047568,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 10300:\tlearning_rate=0.000117,\tloss_train=0.000223,\tloss_val=0.048713,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 10400:\tlearning_rate=0.000117,\tloss_train=0.000000,\tloss_val=0.042749,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 10500:\tlearning_rate=0.000116,\tloss_train=0.000005,\tloss_val=0.038871,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 10600:\tlearning_rate=0.000115,\tloss_train=0.000074,\tloss_val=0.030597,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 10700:\tlearning_rate=0.000114,\tloss_train=0.000003,\tloss_val=0.041348,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 10800:\tlearning_rate=0.000114,\tloss_train=0.000746,\tloss_val=0.034549,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 10900:\tlearning_rate=0.000113,\tloss_train=0.000008,\tloss_val=0.032043,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 11000:\tlearning_rate=0.000112,\tloss_train=0.000000,\tloss_val=0.033110,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 11100:\tlearning_rate=0.000112,\tloss_train=0.000002,\tloss_val=0.053547,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 11200:\tlearning_rate=0.000111,\tloss_train=0.000012,\tloss_val=0.056942,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 11300:\tlearning_rate=0.000111,\tloss_train=0.000033,\tloss_val=0.058680,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 11400:\tlearning_rate=0.000110,\tloss_train=0.000013,\tloss_val=0.059468,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 11500:\tlearning_rate=0.000110,\tloss_train=0.000600,\tloss_val=0.052409,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 11600:\tlearning_rate=0.000109,\tloss_train=0.000304,\tloss_val=0.050763,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 11700:\tlearning_rate=0.000109,\tloss_train=0.000391,\tloss_val=0.043252,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 11800:\tlearning_rate=0.000108,\tloss_train=0.000051,\tloss_val=0.043646,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 11900:\tlearning_rate=0.000108,\tloss_train=0.000001,\tloss_val=0.045690,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 12000:\tlearning_rate=0.000107,\tloss_train=0.000022,\tloss_val=0.036975,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 12100:\tlearning_rate=0.000107,\tloss_train=0.001141,\tloss_val=0.028697,\tacc_train=1.000000,\tacc_val=0.995238\n",
      "\n",
      "\n",
      "Iteration 12200:\tlearning_rate=0.000107,\tloss_train=0.002605,\tloss_val=0.037994,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 12300:\tlearning_rate=0.000106,\tloss_train=0.000231,\tloss_val=0.038053,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 12400:\tlearning_rate=0.000106,\tloss_train=0.000008,\tloss_val=0.041612,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 12500:\tlearning_rate=0.000106,\tloss_train=0.000435,\tloss_val=0.046244,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 12600:\tlearning_rate=0.000106,\tloss_train=0.000046,\tloss_val=0.046227,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 12700:\tlearning_rate=0.000105,\tloss_train=0.000006,\tloss_val=0.042901,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 12800:\tlearning_rate=0.000105,\tloss_train=0.000022,\tloss_val=0.041342,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 12900:\tlearning_rate=0.000105,\tloss_train=0.000032,\tloss_val=0.041868,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 13000:\tlearning_rate=0.000105,\tloss_train=0.000060,\tloss_val=0.041549,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 13100:\tlearning_rate=0.000104,\tloss_train=0.000057,\tloss_val=0.031410,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 13200:\tlearning_rate=0.000104,\tloss_train=0.000524,\tloss_val=0.035047,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 13300:\tlearning_rate=0.000104,\tloss_train=0.000002,\tloss_val=0.036295,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 13400:\tlearning_rate=0.000104,\tloss_train=0.000502,\tloss_val=0.042062,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 13500:\tlearning_rate=0.000104,\tloss_train=0.000056,\tloss_val=0.041174,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 13600:\tlearning_rate=0.000103,\tloss_train=0.000025,\tloss_val=0.038606,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 13700:\tlearning_rate=0.000103,\tloss_train=0.000005,\tloss_val=0.042296,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 13800:\tlearning_rate=0.000103,\tloss_train=0.000003,\tloss_val=0.045483,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 13900:\tlearning_rate=0.000103,\tloss_train=0.000003,\tloss_val=0.047729,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 14000:\tlearning_rate=0.000103,\tloss_train=0.000010,\tloss_val=0.047209,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 14100:\tlearning_rate=0.000103,\tloss_train=0.000014,\tloss_val=0.045859,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 14200:\tlearning_rate=0.000102,\tloss_train=0.000199,\tloss_val=0.050122,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 14300:\tlearning_rate=0.000102,\tloss_train=0.000343,\tloss_val=0.041970,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 14400:\tlearning_rate=0.000102,\tloss_train=0.000013,\tloss_val=0.042751,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 14500:\tlearning_rate=0.000102,\tloss_train=0.000017,\tloss_val=0.047536,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 14600:\tlearning_rate=0.000102,\tloss_train=0.001420,\tloss_val=0.038829,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 14700:\tlearning_rate=0.000102,\tloss_train=0.000053,\tloss_val=0.042733,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 14800:\tlearning_rate=0.000102,\tloss_train=0.000122,\tloss_val=0.036415,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 14900:\tlearning_rate=0.000102,\tloss_train=0.000097,\tloss_val=0.034187,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 15000:\tlearning_rate=0.000102,\tloss_train=0.000019,\tloss_val=0.036098,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 15100:\tlearning_rate=0.000102,\tloss_train=0.000010,\tloss_val=0.053080,\tacc_train=1.000000,\tacc_val=0.995238\n",
      "\n",
      "\n",
      "Iteration 15200:\tlearning_rate=0.000102,\tloss_train=0.000594,\tloss_val=0.047981,\tacc_train=1.000000,\tacc_val=0.995238\n",
      "\n",
      "\n",
      "Iteration 15300:\tlearning_rate=0.000101,\tloss_train=0.000234,\tloss_val=0.046626,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 15400:\tlearning_rate=0.000101,\tloss_train=0.000383,\tloss_val=0.048023,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 15500:\tlearning_rate=0.000101,\tloss_train=0.000006,\tloss_val=0.052817,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 15600:\tlearning_rate=0.000101,\tloss_train=0.006560,\tloss_val=0.060489,\tacc_train=0.990000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 15700:\tlearning_rate=0.000101,\tloss_train=0.000015,\tloss_val=0.044697,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 15800:\tlearning_rate=0.000101,\tloss_train=0.000000,\tloss_val=0.040762,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 15900:\tlearning_rate=0.000101,\tloss_train=0.000000,\tloss_val=0.040853,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 16000:\tlearning_rate=0.000101,\tloss_train=0.000048,\tloss_val=0.040024,\tacc_train=1.000000,\tacc_val=0.995238\n",
      "\n",
      "\n",
      "Iteration 16100:\tlearning_rate=0.000101,\tloss_train=0.000001,\tloss_val=0.041087,\tacc_train=1.000000,\tacc_val=0.995238\n",
      "\n",
      "\n",
      "Iteration 16200:\tlearning_rate=0.000101,\tloss_train=0.000482,\tloss_val=0.047162,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 16300:\tlearning_rate=0.000101,\tloss_train=0.000048,\tloss_val=0.038735,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 16400:\tlearning_rate=0.000101,\tloss_train=0.000004,\tloss_val=0.040919,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 16500:\tlearning_rate=0.000101,\tloss_train=0.000111,\tloss_val=0.028003,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 16600:\tlearning_rate=0.000101,\tloss_train=0.000004,\tloss_val=0.026905,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 16700:\tlearning_rate=0.000101,\tloss_train=0.000041,\tloss_val=0.018132,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 16800:\tlearning_rate=0.000101,\tloss_train=0.000007,\tloss_val=0.019089,\tacc_train=1.000000,\tacc_val=0.995238\n",
      "\n",
      "\n",
      "Iteration 16900:\tlearning_rate=0.000101,\tloss_train=0.000041,\tloss_val=0.024353,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 17000:\tlearning_rate=0.000101,\tloss_train=0.000008,\tloss_val=0.021087,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 17100:\tlearning_rate=0.000101,\tloss_train=0.000031,\tloss_val=0.019918,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 17200:\tlearning_rate=0.000101,\tloss_train=0.000012,\tloss_val=0.021076,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 17300:\tlearning_rate=0.000101,\tloss_train=0.004532,\tloss_val=0.034122,\tacc_train=0.990000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 17400:\tlearning_rate=0.000100,\tloss_train=0.000005,\tloss_val=0.033327,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 17500:\tlearning_rate=0.000100,\tloss_train=0.000002,\tloss_val=0.035994,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 17600:\tlearning_rate=0.000100,\tloss_train=0.000011,\tloss_val=0.035315,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 17700:\tlearning_rate=0.000100,\tloss_train=0.000013,\tloss_val=0.049643,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 17800:\tlearning_rate=0.000100,\tloss_train=0.000011,\tloss_val=0.055924,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 17900:\tlearning_rate=0.000100,\tloss_train=0.000070,\tloss_val=0.053377,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 18000:\tlearning_rate=0.000100,\tloss_train=0.000005,\tloss_val=0.054083,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 18100:\tlearning_rate=0.000100,\tloss_train=0.000000,\tloss_val=0.049477,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 18200:\tlearning_rate=0.000100,\tloss_train=0.000001,\tloss_val=0.048280,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 18300:\tlearning_rate=0.000100,\tloss_train=0.000000,\tloss_val=0.050973,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 18400:\tlearning_rate=0.000100,\tloss_train=0.002151,\tloss_val=0.050878,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 18500:\tlearning_rate=0.000100,\tloss_train=0.004797,\tloss_val=0.060612,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 18600:\tlearning_rate=0.000100,\tloss_train=0.001980,\tloss_val=0.051940,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 18700:\tlearning_rate=0.000100,\tloss_train=0.000092,\tloss_val=0.037035,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 18800:\tlearning_rate=0.000100,\tloss_train=0.000000,\tloss_val=0.036092,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 18900:\tlearning_rate=0.000100,\tloss_train=0.000002,\tloss_val=0.036987,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 19000:\tlearning_rate=0.000100,\tloss_train=0.000039,\tloss_val=0.035656,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 19100:\tlearning_rate=0.000100,\tloss_train=0.022560,\tloss_val=0.030500,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 19200:\tlearning_rate=0.000100,\tloss_train=0.000002,\tloss_val=0.046482,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 19300:\tlearning_rate=0.000100,\tloss_train=0.000039,\tloss_val=0.055406,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 19400:\tlearning_rate=0.000100,\tloss_train=0.000081,\tloss_val=0.041961,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 19500:\tlearning_rate=0.000100,\tloss_train=0.000001,\tloss_val=0.042968,\tacc_train=1.000000,\tacc_val=0.988095\n",
      "\n",
      "\n",
      "Iteration 19600:\tlearning_rate=0.000100,\tloss_train=0.000026,\tloss_val=0.034487,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 19700:\tlearning_rate=0.000100,\tloss_train=0.000002,\tloss_val=0.035798,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n",
      "Iteration 19800:\tlearning_rate=0.000100,\tloss_train=0.000000,\tloss_val=0.039822,\tacc_train=1.000000,\tacc_val=0.992857\n",
      "\n",
      "\n",
      "Iteration 19900:\tlearning_rate=0.000100,\tloss_train=0.000042,\tloss_val=0.041927,\tacc_train=1.000000,\tacc_val=0.990476\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "history = {'acc_train':list(),'acc_val':list(),\n",
    "           'loss_train':list(),'loss_val':list(),\n",
    "          'learning_rate':list()}\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    for i in range(20000):\n",
    "        batch = ch.next_batch(100)\n",
    "        sess.run(train, feed_dict={X: batch[0], Y_true: batch[1], step: i, drop_rate: 0.25})\n",
    "        \n",
    "        # PRINT OUT A MESSAGE EVERY 100 STEPS\n",
    "        if i%100 == 0:\n",
    "            \n",
    "            # Test the Train Model\n",
    "            feed_dict_train = {X: batch[0], Y_true: batch[1], drop_rate: 0.25}\n",
    "            feed_dict_val = {X:ch.test_images, Y_true:ch.test_labels, drop_rate: 0}\n",
    "\n",
    "            matches = tf.equal(tf.argmax(Y,1),tf.argmax(Y_true,1))\n",
    "            acc = tf.reduce_mean(tf.cast(matches,tf.float32))\n",
    "            history['acc_train'].append(sess.run(acc, feed_dict = feed_dict_train))\n",
    "            history['acc_val'].append(sess.run(acc, feed_dict = feed_dict_val))\n",
    "\n",
    "            history['loss_train'].append(sess.run(cross_entropy, feed_dict = feed_dict_train))\n",
    "            history['loss_val'].append(sess.run(cross_entropy, feed_dict = feed_dict_val))\n",
    "            \n",
    "            history['learning_rate'].append(sess.run(lr, feed_dict = {step: i}))\n",
    "            print(\"Iteration {}:\\tlearning_rate={:.6f},\\tloss_train={:.6f},\\tloss_val={:.6f},\\tacc_train={:.6f},\\tacc_val={:.6f}\"\n",
    "                  .format(i,history['learning_rate'][-1],\n",
    "                          history['loss_train'][-1],\n",
    "                          history['loss_val'][-1],\n",
    "                          history['acc_train'][-1],\n",
    "                          history['acc_val'][-1]))\n",
    "            \n",
    "            print('\\n')\n",
    "        \n",
    "    saver.save(sess,'models_saving/my_model.ckpt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 282
    },
    "colab_type": "code",
    "id": "s5rC6lSIAR-P",
    "outputId": "1c59d87f-9158-4a6d-8f25-151c9890c407"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f18508a9748>]"
      ]
     },
     "execution_count": 23,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAHvtJREFUeJzt3XucVWWh//HPvsydAQYcbgoIBY8X\nSgNUULmLIh41b9VJKzvW+XXCXppp2ckyy/LXRfHSRc0OZR2rn5mEeYlEuSSigIgh9GggigIyXIaZ\nYS77tn5/7Av7NsOecfbMPPV9v16+nL332mt/Z7HmO888a629fZ7nISIi7vL3dgAREXlvVOQiIo5T\nkYuIOE5FLiLiOBW5iIjjgj39gnV1jV0+TaamppIDB5q7M0636avZlKtz+mou6LvZlKtzupqrtrba\n195jTo3Ig8FAb0doV1/Nplyd01dzQd/NplydU4xcThW5iIjkUpGLiDhORS4i4jgVuYiI41TkIiKO\nK+j0Q2PMBOCPwEJr7Y+yHjsL+C4QBZ6w1n6721OKiEi7jjgiN8ZUAfcAy9pZ5G7gEuAM4GxjzAnd\nF09ERI6kkBF5GzAf+Er2A8aYscB+a+2OxO0ngDnA5u4M2RWHDsFPf1rKhRdGGDcuBsCGDX4WLSol\nGo0vM3JkjGuvDVFeHr99//0lbNyY/xzPadMifOxjEQAOHoSFC8uoqzt8fn55ObS2lrebx+eDj388\nzNSpURYtKmH9+p45x7W9XLNnR7jkkghPPRXgscdKuvU158+PcN55ERYvDvKXv+TfxY60vbpTWZnH\n5z4XZvToGHfdVcqbb7Y/funJXJ3VV7N1lGvgQI9rrgkRCMCdd5ayf3+717S8J/lep69tL78frrwy\nxLx53b/uIxa5tTYCRIwx+R4eBtSl3d4DvK+j9dXUVL6nE+Jra6sLWu7mr7TSb9FdfP03F7H41XE0\nNcHHPw6+fXUEiLKHIXiU4Hll3HEHbN0KN90EI3mLq/kRK5nOUs4mTCkAS5aU8MlPQv/+sGAB/OHh\nCFECQPqOmVuIE1nPR/kdT3Iun3x8OtfdEOCH32zk33iUSaxnOTMzXqcrymiljbKsLAAefqLE8uRa\n9nADJ/7+R3xz+ZVsi44uaN0+Yqm1duQPfyjh1lvh61+NEMnaRtU0UEkzDfSnhcq8z6+iiX40dbhM\nZ61eXcq8efCTn+R/vJJDVNOYcV8bZdRTk7pdQohB7G/3NWL4qaOW3H+HbB4VtGR8bxU0M58nOJmX\n+SWf4h+Ma+e58X9LP1FqqcPDxx6GAL7UdkvXQgUNDAA8BrOPIJEjZOucEsLM4lnm8Bx+YqxgBv/L\n5aRvg61bSykrgzVP7qeUEADVNHI+j3E8W4gQ5NdcwWrOoB+NVHGIQ1TRxOGf9QHUcwFL+AB/42nO\n4hU+iJe1nfdvhrIyWPZMt36L7creP9rjI0YpIdooZ/z4EubNK7zHCuUr9IMljDHfBPamz5EbY04H\nbrDWXpS4/RlgrLX2v9tbz3u5RL+2tpq6uswftpYWeOSREj784TD9+sXve/zxIG99+ja+wbfZyliu\nnfYi+5or+dL6K7iEPwAQqh3Ob8KXcn/9R7nh56NZ92o/vnPHAN4YO4tjt60A4NCEybz+syf55e+q\nufPOUu68s5WSSAuHrr+NL/ruJDJqDAfOvpiGafOofdvS9tYOGqbMpnzb3yndtYO20e9n5G1fJHAo\nnnk3Q9nISUxnJRW0pr6HtmPG8NZN99AyfgJVr66nes0z+EIhwkOGUz/7AtrGJH6JhsNQUoIv1MaQ\nB+8mPHQEgaYGhv/oFlrGncjuz95I5eaXiNQchVdWzog7b8LvxTgw63zq515M4+TpEAzyxlbo95FL\nONtbypuM4o2rb+O4Qy/RctxJRGqOovqFZ2kdexwl777D8Hu/w6EPnkbD6Wcx7IHvE62qpv6sD1M/\n+wJK9u6m38trIBaj7dhxHJwxn7+/Xc2Dn9vAZTzMBSwhcNRAdn9jIc0nTGTIg3cx5Nf34IvF8MrK\nOHj6XOrPuojQ0KPp/9xSAk2NlO3YRvXa5fiiUaIVVWz/3i9pOPMcyrduYcDyPxGtqqZh+rlEK6qo\neuUFql9ciS8SBsALBmmaPI1YZT+qV/8Ff2t8G2/e4mfTpvjgoX9/j7PnRvCn/S4q3fUW1WuewZ9Y\nT7pDEybTfPzJBA8eiGc81JizTLq2Y8bQeMoMvGD+v3J8kTD91q2ifMdWmk6eQsu4CQT319H/ub8Q\naI1fth0rr2D/vMuIVVSlvp9+L64gcMpp1Af7MfDpxQxctpiSfXvi+/KwkbSNHEu/Dc/hi+QWddNJ\npxFoqKfiDdth9u7SOHk6rYl9ds2aADt2+DiVF5nM+naf4/l8NJ8wkcotG+L7RyBA4ykzaBv5Pkp3\n74j/+4RDPZK/M/bP/xg7r74ZYjEGrHqKQGM9jafNosL+jYrXN+EPtVL9/DJK9r1L0+TphG+9hUFn\nTcvpsUJ0dIn+ey3yY4HfWGunJm7fDOzLPiCarruL/Npry3jooVKuv76NL385xLtvhfjs7N082zCJ\nUn8EfyzKKs4kSoCZrCA84YPERo2mZPUq/PX1qfW0Ucqf/BdySexhQtNm4FVVUfbUE7TNPx9v0xZa\n3trLhn7T+GDT8xzFXkK1wylpqsfX0tJhZi8Q4NBNtxDY/gah3y5hQFsd+4cdT9kVFxKefCplTz1O\n+YOL8MVi7a6j9eLLiI4bT+WdPyQ0ey6EQ5Q9vfTwa1RU5M3hVVbi698fdu8GIDZoEKG58/A1NlL2\nxGNsZSzvY1vH+dPWHRs4ECBju3XkQP+RDGzelVEu0VHHEv7QRMr/YeHVV/M+L3zSh4geO4aypU9C\nNEr02DEEXytuCUVOmEBk/HjKy0pobYsXun/vXkqefw5fYi4uOnIU4Q9NAn/+nydfSwulq1biaz7U\n4Wt5lZVExhuCG1/Gl/j5i4wZS9uFFxMbOYqq796Cf9++DtcRGzSI8NQzIRql5LlV+Bsb4tttzJiM\n5fy7d1PywvNQVkbojGl41d07EgQonzyRA1NnAFD17ZspXZ47JA4TpO20MwkOH5QIFiA89QzCU8/A\nv/Md+t34JYLbthKeNJnoyFEE3txOyYaXUs+PnPgB2i74MOGTTqb02WX4392dsf62Nh/PPhPA82DW\n7Cjl5V7Gv2UxBF97jeDmTUdcLjZgINGRoyjZ9ArN13yJyjt/2LeKPHH/q8B5wNvA88Dl1trX2ltP\ndxR5OAwrVgTYvDnAmltX8hjn82rZRCacVk5g5UqCxH/wGn7yM0p/9SDlz68CoG36LBp+9VuoqIBw\nmJJVy/nH95/grZf2cxovcDQ78crK2L/yBWJDh1Fz7myCWzbj+f3sDQylNryL3Qxl19lXcMx914Pn\nUfb0nylZ8SwVk07mYP/BlK54luixY4mOGUvps8sInT2P0DnnAhALRdj58l6OOXVYxvcVXPciFQ/c\nh6+1lejIUYTOPY9YzSCCm16h4oF7Uzu0V1aGr60NgNCsOYSnnoGvvp7m//oCJevXUrriGcKnn4l/\n9y78b26n5bP/xeBJE6h//C+ULXmU0sf+SGDPu0C8lF7+2QrG/e2PBF/7O+FpMwluWIe/oYHQ9FkE\nN72Cr62N5s9/gdLlzxBc9yLN11yPN2gQJauWU/bnJ4kNGkxo7jl4ZeWUrFkdL71IhMiYsbwx6SKG\nnvchAq9uovInd+NrbiZy3HE0f+E6qKqitraa/X9dS9mSR/HX7SE09xyiI0fjDRhAbPgIAEpWLmfA\nJz4Knkdo9lzazr8QX3196nWiY8bSds58vMQvGN/Bg/Hyb2sldPa5xIYMTW3jtjY4eNDHkCG5u59X\nXU3s6GMy9rEk34H9+N99F6+0lNiYsfGDHR1paSHw5vYOF4mOGg2Vlfj27sW/tw6vvJzY6GMPr7up\nicDbOzK+n/CZMxjw2iZadu2hbd55hM+YBsFg6pvzNTTg1dbmfT3fgf14JaWk/mTtZhnbzPPwv7EN\nX+jw6LmxEUJHDWPgmA6mISIRfE2NeAMPL+PfvQtffT1ev37Ejhl5xBz19RCN+hg82MvNVQyRSPzn\nc83zEAgQmno63uCjKFm5nOj44widOR1KS4mOGQulpfj278Or7k/tiEE9X+TGmEnA7cCxQBh4B1gC\nvGGtfdQYMx34XmLxR6y1P+xofd1R5L/+dQnXXVdOGa1s9p3IaG87AAFirGMS0aOGMuHfj6P5pvif\nPIEtm8HvJ3rc8WT8TQ3EYnDZZRW8sqqJ5877FiPPn0DbxZcB4N+2lcp7FtL6iSt54OXT+MmN73LK\nxcP58b25v+WLttNEIlT8/D78b++g+bovU7ZkMYF/vM6hr91M6ihtBzJyRaMEXrP4ImEi7xsHld0z\n/9wVhW4vX10dVJTj9ev+kWQ+Rf/hfw/6ajbl6pyu5uqWEXl3ea9FHpozl+0bGvjkgbu4Y/KvOX3d\nj/n7uVcz7clvUEqIcO1wli9vpra28Jc5eBD++tcg8+dH2h1wRSLwxBNB5s6NUFGRP9s/005TbMrV\neX01m3J1TjGKvMffj/y9Kn3macYDa5gK6yA6bDgDFt6I9+Jgdu7z89t7OlfiAAMGwHnndXw0PxiE\nCy7o3iP+IiLdwbkij1b1J3CogeeHXsAH/+MkWj92OcFB/bn33lb27vUxe3a0tyOKiPQo94o8EuNl\nJrLs6t/x/v9zeK56xgwVuIj8a3LuTbO8SIwYfqZMUXGLiICLRR6Lgd/PiSe2f961iMi/EueK3OfF\nKCnzpU6hFRH5V+dckfuJ4fmciy0iUjTONaKKXEQkk3ONGFCRi4hkcKsRE1ehHultVEVE/pW41YiJ\ndwjUiFxE5DC3GlFFLiKSw61GVJGLiORwqhG9qIpcRCSbU40Yi6jIRUSyOdWI0bCKXEQkm1ONmBqR\n+52KLSJSVE41Yiwcf8dDjchFRA5zqhGTI3JU5CIiKU41YrLIY5paERFJcaoRddaKiEgupxpRUysi\nIrmcakSNyEVEcjnViDr9UEQkl1ONmLxEX1MrIiKHOdWIGpGLiORyqhF1sFNEJJdTjaiDnSIiuZxq\nxNQcud/Xu0FERPoQR4vcqdgiIkXlVCPqbWxFRHI51YipTwjSiFxEJMWpRtTUiohILqcaUacfiojk\nChaykDFmITAF8IBrrLVr0x5bAFwBRIF11tprixEUIKYRuYhIjiM2ojFmBjDOWjsVuAq4O+2x/sAN\nwDRr7ZnACcaYKcUK6+k8chGRHIU04hxgMYC1dgtQkyhwgFDiv37GmCBQCewvRlBImyMPqMhFRJIK\nmVoZBqxPu12XuK/BWttqjLkF2Aa0AL+11r7W0cpqaioJBgNdCvtuYkReUlpCbW11l9ZRTH0xEyhX\nZ/XVXNB3sylX53R3roLmyLOkLqtMjMz/GxgPNADPGGNOstZubO/JBw40d+El45Ij8nA0Rl1dY5fX\nUwy1tdV9LhMoV2f11VzQd7MpV+d0NVdH5V/IHMVO4iPwpBHArsTXxwPbrLV7rbUhYBUwqdMJC6TT\nD0VEchXSiEuBSwGMMROBndba5K+T7cDxxpiKxO3JwOvdHTIpdfqhilxEJOWIUyvW2tXGmPXGmNVA\nDFhgjLkSOGitfdQY8wPgWWNMBFhtrV1VrLAakYuI5Cpojtxae2PWXRvTHrsPuK87Q7UrGo3/X0Uu\nIpLiVCOm3msl0LWzXkRE/hk5VeTJOXKfRuQiIilONWJqRK4rO0VEUpxqRC+mTwgSEcnmVJGTGJH7\ndIm+iEiKU42o88hFRHI51YiHp1acii0iUlRuNaLe/VBEJIdTjZj8YAmdfigicphbjahL9EVEcjjV\niHqvFRGRXG41Yix5+qHOIxcRSXKqyDUiFxHJ5VQjqshFRHK51YgxXdkpIpLNqUb0dIm+iEgOpxpR\nUysiIrncakRNrYiI5HCrEWO6RF9EJJtTjejpEn0RkRxONaKnC4JERHI4VeR6rxURkVxuNWJiRO4P\nuhVbRKSY3GrEaDT+f43IRURS3GpEnbUiIpLDrUbU1IqISA6nGlGX6IuI5HKrEfXhyyIiOdxqRF2i\nLyKSw61G1AVBIiI5HC1yt2KLiBSTW42oIhcRyeFWI6rIRURyuNWIKnIRkRzBQhYyxiwEpgAecI21\ndm3aYyOB3wClwEvW2s8VIyigIhcRyeOIjWiMmQGMs9ZOBa4C7s5a5HbgdmvtqUDUGDOq+2MmeCpy\nEZFshTTiHGAxgLV2C1BjjOkPYIzxA9OAJYnHF1hr3ypSVny6RF9EJEchUyvDgPVpt+sS9zUAtUAj\nsNAYMxFYZa39akcrq6mpJBgMdCls8oMlBg6qora2ukvrKKa+mAmUq7P6ai7ou9mUq3O6O1dBc+RZ\nfFlfHw3cBWwHHjfGnGetfby9Jx840NyFl0y8WKLIG5paqatr7PJ6iqG2trrPZQLl6qy+mgv6bjbl\n6pyu5uqo/AuZo9hJfASeNALYlfh6L/CmtXartTYKLANO7HTCQmmOXEQkRyGNuBS4FCAxfbLTWtsI\nYK2NANuMMeMSy04CbDGCQvocuS7RFxFJOuLUirV2tTFmvTFmNRADFhhjrgQOWmsfBa4FfpE48Pk3\n4LGipdXBThGRHAXNkVtrb8y6a2PaY/8AzuzOUO3S1IqISA6nGtGnC4JERHK41YiaWhERyeFUI/o0\ntSIiksOtRvR01oqISDanitwXiwKaWhERSedUI+q9VkREcjnViMk5cgJde68WEZF/Rk4VuebIRURy\nOVXkmloREcnlVCP6PBW5iEg2pxpRRS4iksutRkwe7PS7FVtEpJicakRd2SkiksupRvRpRC4iksOp\nRlSRi4jkcqoRk6cfqshFRA5zqhE1IhcRyeVUI/pIFLlPV3aKiCS5VeRejKhbkUVEis6pVvR5MWJu\nRRYRKTqnWlFFLiKSy6lWVJGLiORyqhX9XgzPrcgiIkXnVCv6vBgxn1ORRUSKzqlW9KGpFRGRbE61\noqZWRERyOdWKmloREcnlVCtqakVEJJdTrRjwongakYuIZHCqFTUiFxHJ5VQrxufIA70dQ0SkT3Gq\nyP3orBURkWxOtaJfZ62IiORwqhV9GpGLiOQIFrKQMWYhMAXwgGustWvzLHMbMNVaO7NbE6bxEyOq\nEbmISIYjtqIxZgYwzlo7FbgKuDvPMicA07s/Xia/F9PphyIiWQppxTnAYgBr7RagxhjTP2uZ24Gv\ndXO2HJpaERHJVcjUyjBgfdrtusR9DQDGmCuBFcD2Ql6wpqaSYLBrpxDuIYbn91NbW92l5xebcnWO\ncnVeX82mXJ3T3bkKmiPPkvrkY2PMIODTwFnA0YU8+cCB5i68ZJw/cUFQXV1jl9dRLLW11crVCcrV\neX01m3J1TldzdVT+hcxT7CQ+Ak8aAexKfD0bqAVWAY8CExMHRrud5yXOI/f5jrywiMi/kEKKfClw\nKYAxZiKw01rbCGCt/b219gRr7RTgIuAla+0XixH0cJFrjlxEJN0RW9FauxpYb4xZTfyMlQXGmCuN\nMRcVPV2aaFRFLiKST0Fz5NbaG7Pu2phnme3AzPceKb9YTJfoi4jk40wrpopcI3IRkQzOtKKmVkRE\n8nOmFVMHO/3ORBYR6RHOtKLmyEVE8nOmFTVHLiKSnzOtGI2AH09TKyIiWZxpxVjUA9CIXEQkizOt\n6EVj8f+ryEVEMjjTirGIilxEJB9nWjFZ5KjIRUQyONOKXiQa/78OdoqIZHCmFTW1IiKSnzOtmDrY\nqRG5iEgGZ1pRI3IRkfycaUUvcR65DnaKiGRyphU1tSIikp8zrRgN6/RDEZF8nGnF5NSKRuQiIpnc\nacWYRuQiIvk404rJqRWNyEVEMjnTismDnfh8vRtERKSPca/INSIXEcngTCvqYKeISH7OtOLhqRVn\nIouI9AhnWlFTKyIi+TnTirqyU0QkP2daMfXBEipyEZEMzrSiplZERPJzpxV1ZaeISF7OtOLhEbku\nCBIRSedMkcciifcj19SKiEgGZ1pRc+QiIvk504oqchGR/NxpxZimVkRE8gkWspAxZiEwBfCAa6y1\na9MemwXcBkQBC3zGWhvr7qBeJBr/v4pcRCTDEVvRGDMDGGetnQpcBdydtcj9wKXW2jOAamBet6eE\n1OmHvkCgKKsXEXFVIcPbOcBiAGvtFqDGGNM/7fFJ1tq3E1/XAYO7N2Kc5shFRPIrZGplGLA+7XZd\n4r4GAGttA4AxZjhwNvD1jlZWU1NJMNj5UXVFWQkApRWl1NZWd/r5PUG5Oke5Oq+vZlOuzunuXAXN\nkWfJuSLHGDMEeAz4vLV2X0dPPnCguQsvCc1NLQCEIlHq6hq7tI5iqq2tVq5OUK7O66vZlKtzupqr\no/IvpMh3Eh+BJ40AdiVvJKZZngS+Zq1d2ul0BdL7kYuI5FdIKy4FLgUwxkwEdlpr03+d3A4stNY+\nVYR8KclPCCKgIhcRSXfEEbm1drUxZr0xZjUQAxYYY64EDgJ/Bj4JjDPGfCbxlIestfd3d9DkiNyn\ng50iIhkKmiO31t6YddfGtK/Lui9OB5LvfqgRuYhIBmdaUSNyEZH83GnFmD7qTUQkH2daMTUi19SK\niEgGd1oxeYm+PlhCRCSDc0WuS/RFRDI504pjRsff/XDocI3IRUTSOVPk498fAWDQUSpyEZF0zhS5\nplZERPJzpxVV5CIiebnTil78vVZ0HrmISCZnWtGXHJH7NEcuIpLOmSLX1IqISH7utKKKXEQkL3da\nUUUuIpKXO62oIhcRycudVozFr+zUWSsiIpmcacXDZ604E1lEpEe404qpTwgK9G4OEZE+xqEiT3z4\nsqZWREQyuNOKqamV3o0hItLXFPThy31BaNYc2LiO8KRTezuKiEif4kyRR0+cAIsX49U19nYUEZE+\nxZ2pFRERyUtFLiLiOBW5iIjjVOQiIo5TkYuIOE5FLiLiOBW5iIjjVOQiIo7zeYkPNRYRETdpRC4i\n4jgVuYiI41TkIiKOU5GLiDhORS4i4jgVuYiI41TkIiKOc+aDJYwxC4EpgAdcY61d28t5vg9MI74N\nbwMuACYB+xKL/MBa+3gPZ5oJPAy8mrjrb8D3gV8BAWAX8AlrbVsP57oK+ETaXZOBdUAVcChx35es\ntet7MNME4I/AQmvtj4wxI8mznYwxlwPXAjHgfmvtz3sh1yKgBAgDV1hrdxtjwsBzaU+dY62N9mCu\nX5Bnf+8D2+thoDbx8CBgDfBd4j8Lyf2rzlp7WZFzZffDWoq4fzlR5MaYGcA4a+1UY8zxwP8AU3sx\nzyxgQiLPYGAD8AzwVWvtn3orV8IKa+2lyRvGmEXAj621Dxtjvgv8B/DTngyU2Dl/nsgzA/gIcCLw\naWvtpp7MkshQBdwDLEu7+1tkbSdjzIPAN4BTgRCw1hjzqLV2fw/mupX4D/j/M8YsAK4DvgwctNbO\nLEaOAnNB1v6eWK5Xt1d6QRtj/gd44PBDPba98vXDMoq4f7kytTIHWAxgrd0C1Bhj+vdinpVAcoep\nJz6yDPRenA7NBJYkvn4MOKv3ogDxHffbvZyhDZgP7Ey7bya52+k0YK219qC1toX4CPiMHs71eeCR\nxNd1wOAivn578uXKpy9sLwCMMQYYaK19sYiv3558/TCTIu5fTozIgWEc/rMI4jv0MKChN8Ik/oRN\nTglcBTwBRIGrjTHXAXuAq621e3sh3gnGmCXE/6y8BahKm0rZAwzvhUwAGGNOAXYkpgYAvmWMOQrY\nAlyb2JmLzlobASKJDEn5ttMw4vsaWff3WC5r7SEAY0wAWED8LweAcmPMQ8Bo4BFr7R09mSshY3+n\nD2yvNNcQH60nDTPG/B4YQXxk/L9FzJWvH84p5v7lyog8m6+3AwAYYy4k/g91NfH5rxuttbOBl4Fv\n9kKk14mX94XAp4hPZ6T/su7t7fYZ4BeJr+8CbrDWTic+P7igt0Ll0d526pXtlyjxXwHPWGuT0wjX\nA/8JnA1cboyZ3MOxCtnfe2t7lQJnWmufTdy1D/g68O/Ej2V92xhT9AFNVj+k6/b9y5UR+U7iv72S\nRhA/YNBrjDHnAF8D5llrD5I5f7iEHp6HBrDWvgP8LnFzqzFmN3CKMaYiMdo9miP/eVxMM4EvAFhr\nH027/zHgo70RKE1Tnu2Uvd8dTfzgWU9bBLxurb0leYe19t7k18aYZcAHiB9A7hFpv1Dg8P7+e/rG\n9poBpKZUrLWNxLchwF5jzDrgOIrYIdn9YIwp6v7lyoh8KXApgDFmIrAz8Y/TK4wxA4AfAP+WPDBh\njHnEGDM2schMoDcO4l1ujLk+8fUwYCjxHfiSxCKXAE/1dK5EnhFAk7U2ZIzxGWOeNsYMTDw8k17Y\nXlmeJnc7vUD8F+FAY0w/4vOXq3oyVOKshpC19ua0+4wx5qHEdgwmcr3a7kqKkyvf/t7r2yvhFGBj\n8oYxZpYx5o7E11XAycBrxXrxfP1AkfcvZ97G1hjzf4HUn+HW2o1HeEoxs/wn8T8l03eGRcT/hGoG\nmoifkbGnh3NVAw8BA4FS4tMsG4AHgXLgzUSucE/mSmSbBNxqrT03cfsjwFeIzyW+A1xlrW3uwSy3\nA8cSP6XvHeBy4tM+GdvJGHMpcAPx017vKebcaju5hgCtHD4etNla+3ljzPeA2cR/HpZYa7/Tw7nu\nAW4ka3/vA9vrYuL7/V+ttb9LLBckfvaKIX5Swk+ttYvyrbObcuXrh08lMhRl/3KmyEVEJD9XplZE\nRKQdKnIREcepyEVEHKciFxFxnIpcRMRxKnIREcepyEVEHPf/Afdzy0KORyQ5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history['acc_train'],'b')\n",
    "plt.plot(history['acc_val'],'r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 282
    },
    "colab_type": "code",
    "id": "tkToA3CjBxJe",
    "outputId": "b0930a09-31e6-4eeb-b099-035d807097dc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f1844035358>]"
      ]
     },
     "execution_count": 24,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XuYHHWd7/F3VXXPNTOTIekkJFxC\nAvwgEgh3QZAgLAgnyLpBPSsiu7DHPSy6uuDuknXFXWGRg+sioivLo6B4QRTkalYQoiYQIxeBQAi/\nECAk5EKaZHKZe1/q/FHVM93TPTPdk+mZqfXzeh4epqtqqr9dU/PJb751c3zfR0REossd7wJERGTf\nKMhFRCJOQS4iEnEKchGRiFOQi4hEXGys3zCZ3Dvi02RaWxtoa+sczXJGzUStTXVVZqLWBRO3NtVV\nmZHWlUg0OYPNi9SIPBbzxruEQU3U2lRXZSZqXTBxa1NdlalGXZEKchERKaYgFxGJOAW5iEjEKchF\nRCJOQS4iEnEKchGRiFOQi4hEXGSC3H17EyxZAh0d412KiMiEEpkgr334QbjxRmpWPTXepYiITCjD\nXqJvjGkAvgdMB+qA66y1j+TN3wBsAjLhpIuttZtHu1ByD8BIZ4ZeTkTkj0w591q5AHjWWnuTMeZg\n4FfAIwOWOc9a2z7q1eVzw9sMZLNVfRsRkagZNsittffkvTwQeLt65QzBDbtACnIRkQJl3/3QGLMS\nOABYVGL2bcaY2cCTwBJr7aB3OGxtbRjZTWOaGwBoaaqFRFPl3z8GEqqrIqqrchO1NtVVmdGuq+wg\nt9aeaoxZAPzQGHNMXlhfC/wS2Ak8ACwG7h1sPSO9rWRdRy9NwO5dHfQm945oHdWUSDSRVF1lU12V\nm6i1qa7KjLSuocJ/2LNWjDHHG2MOBLDWvkAQ/oncfGvtXdba7dbaNLAUmF9xheUIWyuOWisiIgXK\nOf3w/cDVAMaY6cAk4N3wdYsx5lFjTE247BnAy9UoVD1yEZHSygny24BpxpgVwC+AK4FPGmM+bK3d\nTTAKX2WMeQpIMkRbZd8qVZCLiJRSzlkrXcDHh5h/C3DLaBZVkoJcRKSkyFzZ6SvIRURKikyQ62Cn\niEhpkQtyjchFRAopyEVEIi4yQd6TCkr1MwpyEZF8kQny3/0+DsCbr49zISIiE0xkgry7Nyi1p1sj\nchGRfJEJcscLS80Mej8uEZE/SpEJ8tzBTvXIRUQKRSbI+0bkOmtFRKRAZIJcpx+KiJQWuSBXa0VE\npFBkgtyNaUQuIlJKZIJcrRURkdIiE+SO5wDgK8hFRApEJsj7RuTqkYuIFIhMkPeffqgLgkRE8kUw\nyDUiFxHJN+yj3owxDcD3gOlAHXCdtfaRvPlnAzcAGWCptfa6ahSaC3L1yEVECpUzIr8AeNZaewbw\nUeA/Bsz/BrAYeB9wjjFm3uiWGNCIXESktHIevnxP3ssDgbdzL4wxc4Cd1tpN4eulwFnAK6NcZ/+j\n3jKZUV+1iEiUDRvkOcaYlcABwKK8yTOAZN7r7cDc0SmtkBsLTj/UiFxEpFDZQW6tPdUYswD4oTHm\nGGttqdNHnOHW09raQCzmVVIjAJNaGgGIx1wSiaaKv38sqK7KqK7KTdTaVFdlRruucg52Hg9st9Zu\nsta+YIyJAQmC0fcWglF5zqxw2qDa2jpHVGhndw8Aqe5eksm9I1pHNSUSTaqrAqqrchO1NtVVmZHW\nNVT4l3Ow8/3A1QDGmOnAJOBdAGvtBqDZGDM7DPhFwGMVV1gGnUcuIlJaOUF+GzDNGLMC+AVwJfBJ\nY8yHw/lXAHcDK4B7rLXrqlNpWKqvHrmISL5yzlrpAj4+xPzlwCmjWVQpuvuhiEhpurJTRCTiIhfk\njoJcRKRAZIJcrRURkdIiE+Q62CkiUlpkgjz3YAmNyEVECkUmyHOtFUfnkYuIFIhMkPedtaLWiohI\ngcgEef+IXEEuIpIvMkGuEbmISGkRCvLgYKdG5CIihSIT5DqPXESktOgFuVorIiIFIhPkfa0VBbmI\nSIEIBbnOWhERKSUyQd53+qFG5CIiBSIT5E74nE+NyEVECkUmyHWwU0SktMgEuRdXa0VEpJTIBHnu\nNrYKchGRQsM+sxPAGHMTcHq4/FestT/Pm7cB2ARkwkkXW2s3j26ZuteKiMhghg1yY8yZwFHW2lOM\nMVOA54GfD1jsPGttezUKzPHCSjUiFxEpVE5rZTnwkfDrXUCjMcarXkmluZ5DFkdBLiIygOP75T+o\nwRjzKeB0a+0ledM2AE8Cs8P/L7HWDrrSdDrjx2KV/zuwezc0To6xrvW9zNv5ZMXfLyIScc5gM8rq\nkQMYYy4ELgfOGTDrWuCXwE7gAWAxcO9g62lr6yz3LQu0t0M9Ln42QzK5d0TrqKZEokl1VUB1VW6i\n1qa6KjPSuhKJpkHnlXuw81zgC8AHrbW78+dZa+/KW24pMJ8hgnykXBeyuLhqrYiIFBi2R26MaQG+\nCiyy1u4cOM8Y86gxpiacdAbw8uiX2R/kuiBIRKRQOSPyjwFTgZ8aY3LTlgEvWWvvD0fhq4wxXQRn\ntIz6aBzA84Ig18FOEZFCwwa5tfZ24PYh5t8C3DKaRZWi1oqISGmRubIzF+QakYuIFIpMkDtOGOQo\nyEVE8kUmyCE3Ii//vHcRkT8GEQxyjchFRPJFLsh1sFNEpFCkgtx31CMXERkoUkGu1oqISLHIBbnr\nZ4ZfUETkj0i0gtzx1CMXERkgWkGu88hFRIpEKsh9R2etiIgMFKkg14hcRKRYpIJcI3IRkWLRCnKN\nyEVEikQqyLOOi6sgFxEpEKkg9x1dECQiMlC0ghyNyEVEBopUkKu1IiJSLFJBrtaKiEixch6+jDHm\nJuD0cPmvWGt/njfvbOAGIAMstdZeV41CITz9UCNyEZECw47IjTFnAkdZa08BPgh8fcAi3wAWA+8D\nzjHGzBv1KkPqkYuIFCuntbIc+Ej49S6g0RjjARhj5gA7rbWbrLVZYClwVlUqBXxXQS4iMtCwrRVr\nbQboCF9eTtA+yd1LdgaQzFt8OzB3qPW1tjYQi3kjKBXeDkfkiUTTiL6/2lRXZVRX5SZqbaqrMqNd\nV1k9cgBjzIUEQX7OEIs5w62nra2z3Lcs4jsuHlmSyb0jXke1JBJNqqsCqqtyE7U21VWZkdY1VPiX\nddaKMeZc4AvAedba3XmzthCMynNmhdOqwnfCcn2/Wm8hIhI55RzsbAG+Ciyy1u7Mn2et3QA0G2Nm\nG2NiwCLgsWoUCnlBnlWfXEQkp5zWyseAqcBPjTG5acuAl6y19wNXAHeH0++x1q4b9SpDBUHujazP\nLiLyP005BztvB24fYv5y4JTRLGowvqsRuYjIQJG7shPAT+sBzCIiOREL8qCd4mc0IhcRyYlYkAfl\nZlIKchGRnGgFedgjz6YV5CIiOZEKchwFuYjIQJEK8r6DnRldECQikhOtIHdzQa4RuYhITqSCHB3s\nFBEpEqkg7x+Rq7UiIpITqSDXwU4RkWKRCnL1yEVEikUqyDUiFxEpFqkg14hcRKRYpIIcXdkpIlIk\nWkGu1oqISJFIBblaKyIixSIV5Og8chGRItEKckcjchGRgaIV5LmDnRqRi4j0KefhyxhjjgIeBG62\n1n5zwLwNwCYg9/y1i621m0exxn651ooOdoqI9Bk2yI0xjcCtwBNDLHaetbZ91KoajE4/FBEpUk5r\npQc4H9hS5VqGlTtrhYweviwikjPsiNxamwbSxpihFrvNGDMbeBJYYq0dtInd2tpALOZVWmfADb5v\nUkMtiUTTyNZRRROxJlBdlZqodcHErU11VWa06yqrRz6Ma4FfAjuBB4DFwL2DLdzW1jnydwpH5Ht2\ndZBM7h35eqogkWiacDWB6qrURK0LJm5tqqsyI61rqPDf5yC31t6V+9oYsxSYzxBBvk90QZCISJF9\nOv3QGNNijHnUGFMTTjoDeHnfyxpE38FOnX4oIpJTzlkrxwNfA2YDKWPMRcBDwJvW2vvDUfgqY0wX\n8DzVGo1DX5CT1YhcRCSnnIOdzwELh5h/C3DLKNY0OLVWRESKROrKTsfTeeQiIgNFKsj7RuRqrYiI\n9IlkkKPWiohIn2gFuaceuYjIQJEKckf3IxcRKRKpINfphyIixSIV5DprRUSkWKSCXCNyEZFikQry\n3IicrHrkIiI5kQry/ke9aUQuIpITrSD3dB65iMhAkQryXGtFV3aKiPSLVpDr4csiIkUiFeS5Hrnj\nK8hFRHIiFeRO7lmf6pGLiPSJVpDrXisiIkUiFeR6sISISLFIBbkT05WdIiIDDfuoNwBjzFHAg8DN\n1tpvDph3NnADkAGWWmuvG/UqQ44u0RcRKTLsiNwY0wjcCjwxyCLfABYD7wPOMcbMG73yCvVfoq8g\nFxHJKae10gOcD2wZOMMYMwfYaa3dZK3NAkuBs0a3xH66IEhEpNiwrRVrbRpIG2NKzZ4BJPNebwfm\nDrW+1tYGYrnTCCv0VhjkNTGPRKJpROuopolYE6iuSk3UumDi1qa6KjPadZXVI6+AM9wCbW2dI195\nGOS9Xb0kk3tHvJ5qSCSaJlxNoLoqNVHrgolbm+qqzEjrGir89/WslS0Eo/KcWZRowYwW9chFRIrt\nU5BbazcAzcaY2caYGLAIeGw0CitFPXIRkWLDtlaMMccDXwNmAyljzEXAQ8Cb1tr7gSuAu8PF77HW\nrqtSrRqRi4iUUM7BzueAhUPMXw6cMoo1DcrVBUEiIkWidWWnHiwhIlIkUkHe94Qg3cZWRKRPpILc\nDYPcUWtFRKRPpIJct7EVESkWySDH98e3EBGRCSRSQZ47a0WtFRGRfpEKcp1HLiJSLFJB3j8iz4xz\nJSIiE0ekgjz38GVdoi8i0i9SQa4euYhIsUgGuXrkIiL9IhXkuYOdjq7sFBHpE60g7xuR6zxyEZGc\nSAW5F9O9VkREBopUkDu614qISJFIBbkOdoqIFItkkOtgp4hIv0gGuXrkIiL9IhXk6pGLiBQb9pmd\nAMaYm4H3Aj7wWWvtM3nzNgCbgNwNUC621m4e3TIDXlwjchGRgYYNcmPMGcBh1tpTjDFHAndQ/LDl\n86y17dUoMJ8uCBIRKVZOa+Us4AEAa+1aoNUY01zVqgahe62IiBQrp7UyA3gu73UynLYnb9ptxpjZ\nwJPAEmvtoJdetrY2EAvvYlipzj3BoN9zIJFoGtE6qmki1gSqq1ITtS6YuLWprsqMdl1l9cgHcAa8\nvhb4JbCTYOS+GLh3sG9ua+scwVsGmsIReTaTIZncO+L1VEMi0TThagLVVamJWhdM3NpUV2VGWtdQ\n4V9OkG8hGIHnzAS25l5Ya+/KfW2MWQrMZ4gg3xdqrYiIFCunR/4YcBGAMeY4YIu1dm/4usUY86gx\npiZc9gzg5apUSv9ZKzrYKSLSb9gRubV2pTHmOWPMSiALXGmM+Qtgt7X2/nAUvsoY0wU8T5VG46Ar\nO0VESimrR26tvWbApBfz5t0C3DKaRQ1Gpx+KiBSL1JWduLkg18OXRURyohXkXnDaog52ioj0i1aQ\n50bkKMhFRHKiGeTqkYuI9FGQi4hEnIJcRCTiohXkTnB3AB3sFBHpF7kgz+LQ2+2TSo13MSIiE0O0\nghzwHRf8LOvWRa50EZGqiF4aui4uWV56KXqli4hUQ/TSsC/IR3ZPcxGR/2kiF+SO5+KRZfXqyJUu\nIlIVkUtDx3Wpr8vy8sseOnlFRCSCQe67Lg11GTo6HN58c+DDikRE/vhELshxXRrqgqH473+vPrmI\nSOSC3G9oINH2GpfxXR76Sf/J5FdfXcvixfXs3j2OxYmIjIPIBXn7DV/Ficf4Ln/FY6ta8T7+Fzz3\ntM8PflDDihUxLr20nu7u8a5SRGTslPWEoImkd9GHaDtmARuvvo2a3yzjPY//nG1vHA18iQULMqxc\nGeP00xu56qoeTj45w5o1Ht3dcNFF6dwV/kXuuy/GV75Syz0/6WDuXJ+CBf0Br6vE2b0Lv2Vy1d9n\nXGQy1N5/L6TT9Pzvi/smO+178Rsa++6hUyCbxbOv4q1fh9/cQnreUfhTpuBu2kjsNUvmoNlk5syF\nWP8u7L6zjfivn8Db8CZks/hNzWQPOojUSe/F3baVmt/+mviyxyFeQ3rePDo/cxX+lCk4He34zS2j\n93l9H6d9L+72d3B27sRJpcjMmUt2xv6j9x5Rkc0Se3k12aZmsofMGd9aenogHi+9v+Vxkklidi2p\nY4+HxsYxKm7fRC7IAbIHHkTrnTdw+nt6Wdm5gEve+DLN07ew8OJj+EWzi7viKTo+W89pfIs0cWro\nof7HP+KCr56Mf+ihBevq7YXrr6+lbvMbzPqTRTQvNOz5zvfB86j92U+Y9MVr6Pzbq+m69DJqfvck\n3tq1ZA49jN7z/hd0duJ0dOAnEiP+LE4yCZ/5PFOX3U/ygk/gfP1G/KZm6OnBe3093sa3SM97D9np\nM4itXUPsxRcgnaLngg/jT5vW/yE8r//BG7t34W7bRuZw0/9GqVQQnDW1Q++cHR3UPv4o8d8sA6Dz\n89eQnXXA8B/E93E3vkXNE78CP0t6wXFkZ+xP/OlV1N/6deIvrw5Wv2UzmZmz4N67mbp8Oem5h9J7\n7vk4nR1kE9PIHDIHfJ/6275F/KUXC9/CdQvus5OZOYuuyz6FP3kyNY/9NzW/ehTH94cuM1xHzYrf\nUPfjH0LMw21rI/2e+aROfi+YQ2l8fQPunj2QTkM6RXbadNILjiNzuCF96OHQ0ID7zjacXbvIHHZ4\nfzC0tzPpn/+Rugfuw+nsLHrvzIEHkZ0yhYw5ktSJJ0MmEyzn+6TedxrpBccFg4Z0Gqd9L/Sm8DZu\nwFv/Gt5bG6DGpSHjkJ2aIJuYht/c3Lft/cmTSc8/pn/QEf48vDffwG9sxG+ZjF9bi7v9Hbwtm3G3\nbMHdshmnqwu/vg7q6nH27Mbd/Dbpo+aTPuY4nJ5uvPWvQaqX3jP/BACns4P0scfht+4XvE9nJ3zh\nRlp/+jMysw8hO+tA/LpanM5OvLfeIvbSC7htbfiuS/fFn6T37HPJTp0KflAjvo+T6iW25iWc9nZ6\nzzgTf1IT1NaQmXNo0TaslLN3D/W3f5vahx/EW7sGPA9/0iRIpcnuvz+ZQ+bgNzbi7tmDs6sN0pmg\nlkwGv66Ongv/jI6/X0L2wIOKB3S+T83DD1D/w+8Tf3oVvaefQfdH/xxwSB+zgOxBB/dtI2/rZtxk\nkvRR8yHRtM+fq+hz+sPs+ADGmJuB9xJs/s9aa5/Jm3c2cAOQAZZaa68bal3J5N7h33AQiUQTyeTe\nvtcrV3qs+toz/MPKxbRmdhQt//CcT7PppD/lA/dfzRE9L9LutbD2H/+LN6cez9FHdDEjtZkXvv8K\ny3++i8u4gwPYDEDH5z5P+rgTaL78Epzwpi5+LIaTTvetu/Ov/4bapY/gbn+Hjn/+FybNPoD2DW+T\nOuEk/Lp68Dwy5gjo7sbd8S7ZWQdQd9ed1D78IF2Xf4reD56Pt87S8pEL8bZtZTfNtLAH3/PITpse\njOYy/Y+08z2v8HUsRvqIeVBbE4S775OdmsDfbwre+nU4qRQ9Hzyf2ov/nK5lv6X2vp/h7gkOIKQP\nO5yeY05grX8Es4+fTH1yI/FVv8Pp7CT22jqczo6+98k2NePvtx9ORwep955KtqUFXJfUqafhvfE6\nNb/9Ne62rbjbtuL09g76s+te/FHiq1bibX67b1pqwbHE1r6C09NT8nt6Fl1I6sSTcdp2ErOv4r6b\nJJuYRvqo+Xgb3qT2kQcLAjN1/In0fOjDpOcfjR+L4+7ZhffqWuLPPk12/5mkTjyZ3rPPwa+to+7u\nH9J407/h1zeQOXg28WefHrL+vu1eW0v6yHnEVr+Ik82SbW0lc9Bs/IYGvA1v4m3dQubg2aQPN0HY\n7jcFPxYj9vJqYi+txt29C2eQ3l9mxv6kjzuB+O+exG1rG7aWou8/eDbZ5ha8rVtwOtpxuroqXke5\nsq2t+I2TcHfuwOnsxK+rK/m5MgcdTOrU04g9/xwx+2pF79F7+hmk3nc62f2m0HvWn5A94ECc3buI\nvbIGHIfMjP2D0Pd9YuvXUf9f/4m3fh3EYrjvvovb3YXf24vT04NfXx/8Qxf+tYTr4W7a2Pc7AeDX\n1IDjkD5yHqkTT6Zm2ePEXl/fPz8Wg3gc34vhT5lCT7yRhvVrgs85cxbels2Fn33adJyuLty9e/qm\ndV7xGRr+8xsFOVauRKJp0NbAsEFujDkD+Htr7SJjzJHAHdbaU/LmvwKcC2wGfgv8tbX2lcHWN5pB\n3iedJvbsM3gb3sDJZEgfOY+mz11J7NW1fYs8NfUCTnj3UWoZ/Jf1Wv6Vy7iD2bwFQMaNcd1h3+MT\n/IiZmY3YQ8/jx6uP4ap3lrB/5m18xyE7qRlvb+kjrD2TE8Q79+D29pCdPBl3167+kmcdhLNnD97e\nXSzhBm7m77iar3HVkb+gZddGemccSO/hRxI75ADif3gGd+fOcKR0LHR1UnfvPcTWvgK9vaSPPgbi\nNUH4J5P0HDiHHq+ByWtW9b1fZtp0uhecSGbnXiateRa3q6OgVt918WvryM6YQc+HL2LvmefjvLCa\nqf9+LU4shl9bW7SjAmRdj9TUGTizZuAfcCDdpy3Er6ujZs1q3OQ7pKfPYusFf0nd/Lk0bn6Nxi8u\nIX3MsTRe+dckJ03F2bED77V1+C0tuFu34G3aiNPZSeq004NfvCE4O3ZQ8/ijwS+fOZLOIxbQ1RX8\nYTJpUhkdsdy+7zjQ1UXMrqW1o422xlayk1uDtk0shvvWW8RXP4/3xuvEVz5FbO0aUsefQOaQucSe\newZv21acri6yjZPouvQyOpZcix+vKXob3ydoNax5idial/Dr6vDrG3E6O6j79a+oe/yXuLvaSCVm\n0HP0ccTq42QPPIjM3EPJzD6EyftPYdfmJO6Od3GT23H27Alqdxy819dT+9+PgO+TmTkLf1ITmdmH\nkDnc4HR34+zejdPdRXbadDIzZ5LdfxbZmTPxGxpxerqhqxsa6snMmEn86VV4G97Er6sle/BsyGSp\n+c0T+PUN+DVx4n94LhjNd3biN7cQu/ACkp/6W5yebtxkEqe7C7+hMdiXapt58UUXUilOanuMujcs\n7q62vrp9xwHXJXO4wa+ppWb5r4PPs349NSt+U/jjcpxh/+JKNbfiOT7Z/fYj1tJMKp2l94IL6fw/\nVwQ7xYCfv7N3D05nJ9lJTfiNk9i9G5JJl/Z2aG3JcPDKn9Ly8E+CbRj+heb3pulZv5mmnh2snP4h\nDvrRl6idfyjx3z1F7IXnwfeJP7Wc2Gvr8BsnkU0kyMw6gOzMWXR/7ONMOWH+uAT5l4GN1trvhK9f\nBU6y1u4xxswB7rLWnhbOWwK0W2tvHWx9VQnyErw31tP0fy8nc+jhdH/iUnpPOY1Vt75Aw313s1/m\nXda83siWzDRe5BhO/1iCj/5dggdeNqz45isseuFGkkzlXi5iGWcVrtfzmZNdz/X+F/g+l/I8x/JZ\nbmE709jONE7m9/g4TGYXH2AZ25nGWxzMSTzNc5zAzd5VXJH9Ngv9ZTTSwd/yDe5wLuff/72Hq6+u\nK/ocsZhPPF76Mzp+ljgpep3avmnZLPT0ODhkuYh7mRVP8qpzBCuc99PRE6zII81cXufUKa+S2rGX\nNlpZzvtppwnX9YnFoLc3t8/4OA7U1vjMZgNxUjSxl/enl7EpM5MH+FM6CH5B4nGfVKp/X6up8fPW\nE7zuD1eH4A+8wVVyaCKVgkym/xtiMb+vfZ6/iw/3te/31zXYsrV+N93U5727j4OPv4/nDnikmc0G\n3uQQsnh4np/rloUcHKdwm+VvI89Pk3W8MTmmU2jwn2UqBel0UI/n+dTUlFyspMOzrzLLf5u5/nrO\nzzxCE3top4mX3aPIEGN/fyuNfju+47AzO5nv+5/kKU4jFsu9z/D7WL7e3v5a88XjhT+HTCb4XFPr\nO3i3axLxuJ9/qGZQjgNf/GIP11xTNy5BfjvwC2vtg+HrFcDl1tp1xphTCUbrHw7nXQ7Mtdb+02Dr\nS6czfiw2/ud/v/46rFgBZ54JBx9cOG/zZrj7bmhogHPPhTvugI0bYc4cuOQS2L4drr8+aKE2NQUt\n53Qaamrg6KODtmEyCXV1wbo2bQoCNmwJks1Cc5PPlJY023bEOeccuOYa+NKXYNmyYOAwaVKws7zz\nTrDjVGLWLJg5E6yFPf1/1dHSAjNmBOs89tjgM9x0Ezz2GNTWBu/T2xu8b0tL8Pm7uoLPU6obcNhh\nMH8+rF4N27YFy9bVBTtsKhX8V18PU6ZAWxt0dBSvYzBldPwKeF7wc2hsDN53504KrvzNz7bhvq5k\n2dFet+MEP/t0GnbsKPzZD9wmw72eCOJxOC5s/f/hD8H+VQ2xGOQOf61fz4hucx2LwbRpMH168Hu9\nc2fwu75jR/G2PfFEuOEGuPFG+NWvyvu303Xhn/4JPvShymsLjWqQPwlcNkiQ/xUwZ6ggH6sR+Vib\nqLWprspM1Lpg4tamuioz0rqGGpGX87fgFmBG3uuZwNZB5s0Kp4mIyBgpJ8gfAy4CMMYcB2yx1u4F\nsNZuAJqNMbONMTFgUbi8iIiMkWFb9NbalcaY54wxK4EscKUx5i+A3dba+4ErgLvDxe+x1q6rWrUi\nIlKkrAuCrLXXDJj0Yt685cApiIjIuIjcvVZERKSQglxEJOIU5CIiEacgFxGJuLJumiUiIhOXRuQi\nIhGnIBcRiTgFuYhIxCnIRUQiTkEuIhJxCnIRkYhTkIuIRFxZN82aCIZ6APQ41XMTcDrBNvwK8CHg\neCD3FOivWmt/McY1LQR+BqwJJ70E3AT8APAI7iN/ibW29NOOq1fX5cAleZNOAJ4FGoHcc4OuttY+\nN4Y1HQU8CNxsrf2mMeZASmwnY8zFwOcI7vx5u7X2u+NQ151AHEgBn7DWbjPGpICn8r71LGtthc+S\n2qe6vkeJ/X0CbK+fAYlw9n7AKoKHw78E5PavpLX2I1Wua2A+PEMV969IBHn4AOjDrLWn5B4AzTje\ncdEYcyZwVFjPFOB5YBmwxFr7yHjVFfqttfai3AtjzJ3At6y1PzPG3ABcBnx7LAsKd87vhvWcAXwU\neA/wl9bal8eylrCGRuBW4Im+l9HIAAAECklEQVS8yV9mwHYyxtwFXAucBPQCzxhj7rfW7hzDuq4n\n+AX/qTHmSuAq4B8IbiO9sBp1lFkXDNjfw+XGdXvlB7Qx5g7gO/2zxmx7lcqHJ6ji/hWV1spZwAMA\n1tq1QKsxpnkc61kO5HaYXQQjy/F/EGlpC4GHwq8fBs4ev1KAYMe9bpxr6AHOp/BpVgsp3k4nA89Y\na3dba7sIRsDvG+O6/ga4L/w6CUyp4vsPplRdpUyE7QWAMcYAk621T1fx/QdTKh8WUsX9KxIjcoLH\nyeX/2Z0Mp+0pvXh1hX/C5loClwNLgQzwaWPMVcB24NPW2nfHobx5xpiHCP6s/FegMa+Vsh3Yfxxq\nAsAYcyKwKWwNAHzZGDMVWAt8LtyZq85amwbSYQ05pbbTDIJ9jQHTx6wua20HgDHGA64k+MsBoM4Y\n82PgYOA+a+1/jGVdoYL9nQmwvfJ8lmC0njPDGHMvwaMqv2Wt/VEV6yqVD+dWc/+Kyoh8oDKeWV19\nxpgLCX5Qnybof11jrf0A8ALwL+NQ0msE4X0hcClBOyP/H+vx3m5/BXwv/PoWggd3v5/wyVPjVVQJ\ng22ncdl+YYj/AFhmrc21ET4PfAo4B7jYGHPCGJdVzv4+XturBjjNWvvrcNIO4IvAnxMcy7rOGFP1\nAc2AfMg36vtXVEbkQz0AelwYY84FvgB80Fq7m8L+4UOMcR8awFq7GbgnfPm6MWYbcKIxpj4c7Y73\nw7EXAp8BCB8TmPMw8LHxKChPe4ntVOrh4qvGobY7gdestf+am2CtvS33tTHmCWA+wQHkMZH3Dwr0\n7+/3MjG21xlAX0slfMbwneHLd40xzwJHUMUMGZgPxpiq7l9RGZEP+gDo8WCMaQG+CizKHZgwxtxn\njJkTLrIQGI+DeBcbYz4ffj0DmE6wAy8OF1kM/HKs6wrrmQm0W2t7jTGOMeZxY8zkcPZCxmF7DfA4\nxdvp9wT/EE42xkwi6F+uGMuiwrMaeq21X8qbZowxPw63Yyysa82gK6lOXaX293HfXqETyXscpTHm\nTGPMf4RfNwILgKo9W7hUPlDl/Ssyt7E1xtwI9P0Zbq19cZhvqWYtnyL4UzJ/Z7iT4E+oTqCd4IyM\n7WNcVxPwY2AyUEPQZnkeuAuoA94K60qNZV1hbccD11trzwtffxT4R4Je4mbgcmtt5xjW8jVgNsEp\nfZuBiwnaPgXbyRhzEfD3BKe93lrN3uogdU0Duuk/HvSKtfZvjDH/D/gAwe/DQ9bafxvjum4FrmHA\n/j4BttefEez3T1pr7wmXixGcvWIITkr4trX2zlLrHKW6SuXDpWENVdm/IhPkIiJSWlRaKyIiMggF\nuYhIxCnIRUQiTkEuIhJxCnIRkYhTkIuIRJyCXEQk4v4/t6XIQVhYwB8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history['loss_train'],'b')\n",
    "plt.plot(history['loss_val'],'r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 282
    },
    "colab_type": "code",
    "id": "NJPdR4IFb-uc",
    "outputId": "4409d520-8b2b-42fa-a9ac-61875d25abdf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f1843f6c550>]"
      ]
     },
     "execution_count": 25,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xt8XHWd//HXZCb3S5OmadOU0NLS\nfuiFiwUWCmqrrbAqq7sL617wgovrPpDdRf25yu/h46crKLrrruyyN9fHT3BFUVQeuPCzIgoqSLm0\nFYq25VNaWnpJ26RtmqRt7jO/P+akTEMmnUkzOZPM+/l48EjmnPM98z7DdD453+8534kkEglERKSw\nFYUdQEREwqdiICIiKgYiIqJiICIiqBiIiAgQCzvAWLS1dY35Eqi6ugra20+MZ5xxk6/ZlCs7ypW9\nfM021XI1NFRH0q0ruDODWCwadoS08jWbcmVHubKXr9kKKVfBFQMREXk9FQMREVExEBERFQMREUHF\nQEREUDEQERFUDEREhAIrBntbj/HtR14irmm7RUROUVDF4JktB/nuT52dLZ1hRxERySsFVQxm1pUD\n0HLoeMhJRETyS0EVg9n1FQC0HFYxEBFJVWDFoBKA/Yfzb+IpEZEwZTRrqZndCVwOJIBb3H19yro1\nwB3AILDW3W9P18bMVgBfBvqBXuB97t5mZtcDHwXiwNfc/evjdYCpqsqLqasuVTeRiMgwpz0zMLOV\nwEJ3XwHcCNw1bJO7gGuBK4GrzGzJKG0+Drzf3d8CPA38hZlVAp8B1gCrgI+Z2fQzPrI0mmdVc7ij\nh96+wVw9hYjIpJNJN9Fq4IcA7r4VqDOzGgAzmw8ccfc97h4H1gbbj9jG3f/I3V8xswgwB9gLXAas\nd/cOd+8GniJZWHKieVY1CeDAEXUViYgMyaSbqBHYmPK4LVjWGfxsS1nXCiwAZqRrY2a/S/JMYSvw\nLeBPRtjH7NEC1dVVjHk+7+aZVQB09Q3S0FA9pn3kUj5mAuXKlnJlL1+zFUqusXzTWdpvyhll3cnl\n7v6ImRnwJeBWYFcW+wc4o28eam5MvoC+8zDLzq4d835yoaGhmra2rrBjvI5yZUe5spev2aZartEK\nSCbdRC0k/6of0gTsT7NuTrBsxDZm9gcA7p4AHgDeOMo+cqJ5VvLF0CCyiMhrMikGjwLXAZjZcqDF\n3bsA3H0XUGNm88wsBlwTbJ+uzd+Z2UXBfi8DHHgWuNTMas2siuR4wZPjdHyvU1tVSmVZjBZdXioi\nctJpu4ncfZ2ZbTSzdSQv/bzZzG4AOtz9QeAm4DvB5ve7+zZg2/A2wfobgf8wswGgm+Slpd1mdivw\nE5KXoX7O3TvG8RhPEYlEmD2jklf2ddI/EKc4VlC3WoiIjCijMQN3v3XYok0p654AVmTQBnffAFwx\nwvIfAD/IJMt4aKqvYPveDg62n+CshqqJeloRkbxVkH8WNwV3ImvcQEQkqTCLwQxNSyEikqogi8Fs\nnRmIiJyiIIvB9JpSSkui7NfspSIiQIEWg0gkwuzpFRw4coLBeDzsOCIioSvIYgDJcYOBwQRtR3vC\njiIiErqCLQZzgkHkfW3HQk4iIhK+gi0GZwUT1u1pVTEQESncYhDcbLa3TYPIIiIFWwxqq0qoKi9m\nr84MREQKtxhEIhHOaqik9Wg3PX0DYccREQlVwRYDeG3cYJ+6ikSkwBV0MWgOxg326IoiESlwBV0M\nhs4MNG4gIoWuoItB04xKIhEVAxGRgi4GpcVRZtVVsKftOIlEIuw4IiKhKehiAMmuou7eAY509oYd\nRUQkNAVfDJobktNSaBBZRApZwRcDDSKLiKgYnLy8dK/ODESkgBV8MaifVkZZSVQT1olIQSv4YhCJ\nRDhrZhUHj3TTPzAYdhwRkVAUfDEAaJ5ZRTyRoOXQibCjiIiEQsWAZDEAePVgV8hJRETCEctkIzO7\nE7gcSAC3uPv6lHVrgDuAQWCtu9+ero2ZNQP3AMVAP/Bedz9gZv3AUylPudrdJ6zPZu6sagBePdAF\nF07Us4qI5I/TFgMzWwksdPcVZrYYuBtYkbLJXcDVwD7gl2b2ANCQps3nga+5+/fM7Gbg48AngQ53\nXzWOx5WVsxqqiBZF2HVAZwYiUpgy6SZaDfwQwN23AnVmVgNgZvOBI+6+x93jwNpg+3RtPgI8EOy3\nDagfx2MZs+JYEXMaKtnTeoyBwXjYcUREJlwm3USNwMaUx23Bss7gZ1vKulZgATBjpDbuvg3AzKLA\nzcBtwfoyM7sPmAs84O5fGS1QXV0FsVg0g+gja2ioft2y8+bVs/vgMXricE7j69dPlJGy5QPlyo5y\nZS9fsxVKrozGDIaJjGHdyeVBIbgXeNzdHwsWfwL4FsnxhSfM7Al335DuSdrbx37VT0NDNW1tr+8O\nmlVbBsDzWw9QVRzOuHq6bGFTruwoV/byNdtUyzVaAcmkGLSQPAMY0gTsT7NuTrCsb5Q29wAvu/vn\nhla6+1eHfjezx4DzgbTFIBdSB5HfdMFEPrOISPgy+RP4UeA6ADNbDrS4exeAu+8CasxsnpnFgGuC\n7UdsY2bXA33u/tmhnVvSfWYWCfZxJbB53I4wQ80zK4kWRZJXFImIFJjTnhm4+zoz22hm64A4cLOZ\n3UDyCqAHgZuA7wSb3x+MC2wb3iZYfzPJ8YFfBI+3uPtHzGwP8Fyw7UPu/tw4HV/GimNRmmYkB5EH\n43GiRboFQ0QKR0ZjBu5+67BFm1LWPcGpl5qma4O7X5Fm/5/KJEeuzW2sZk/rMfYfOnFyNlMRkUKg\nP39TzAuuItL9BiJSaFQMUsxtTLkTWUSkgKgYpGhuqKIoEtEcRSJScFQMUpQUR2maUcHu1i7i8UTY\ncUREJoyKwTBzG6vp64+z//DxsKOIiEwYFYNh5jXWALBzv7qKRKRwqBgMM78pWQxe2d8ZchIRkYmj\nYjBM88wqimNFvLKvI+woIiITRsVgmFi0KHnzWdsxevv0ncgiUhhUDEawoKmGRAJ2qqtIRAqEisEI\nFjRNA2BHi7qKRKQwqBiMYMGcZDF4pUVnBiJSGFQMRlBXXUpddSk7WjpJJHTzmYhMfSoGaSxoqqHz\neB+HO3rCjiIiknMqBmnMPzluoK4iEZn6VAzSWDAnefPZDt1vICIFQMUgjbmzqokWRXRmICIFQcUg\njZLiKGfPqmL3wS76B3TzmYhMbSoGo5jfNI3BeIJXDx4LO4qISE6pGIxiQTBp3fa9GjcQkalNxWAU\ni5prAdi252jISUREckvFYBTTa8qYMa2Ml/ceJa6bz0RkClMxOA1rruV4zwAtbfrmMxGZulQMTmOo\nq8jVVSQiU1gsk43M7E7gciAB3OLu61PWrQHuAAaBte5+e7o2ZtYM3AMUA/3Ae939gJldD3wUiANf\nc/evj9cBnqlFZ79WDFZffFbIaUREcuO0ZwZmthJY6O4rgBuBu4ZtchdwLXAlcJWZLRmlzedJftiv\nBB4EPm5mlcBngDXAKuBjZjb9jI9snMysLWdaVQnb9hzVpHUiMmVl0k20GvghgLtvBerMrAbAzOYD\nR9x9j7vHgbXB9unafAR4INhvG1APXAasd/cOd+8GniJZWPJCJBLBmmvpPN7HwfbusOOIiOREJt1E\njcDGlMdtwbLO4GdbyrpWYAEwY6Q27r4NwMyiwM3AbWn2MXu0QHV1FcRi0Qyij6yhoTqr7ZcvbuS5\nra20tPdwvs0a8/NmIttsE0W5sqNc2cvXbIWSK6Mxg2EiY1h3cnlQCO4FHnf3x8zsz7LYPwDt7SdO\nGzKdhoZq2tq6smrTVFcGwMYtB1i+IHc9WGPJNhGUKzvKlb18zTbVco1WQDLpJmoh+df7kCZgf5p1\nc4Jlo7W5B3jZ3T93mn3kjaYZlVSWxXTzmYhMWZkUg0eB6wDMbDnQ4u5dAO6+C6gxs3lmFgOuCbYf\nsU1w1VCfu382Zf/PApeaWa2ZVZEcL3hyXI5unBRFIixqruVwZw+HOjRuICJTz2m7idx9nZltNLN1\nJC/9vNnMbgA63P1B4CbgO8Hm9wfjAtuGtwnW3wyUmdkvgsdb3P0jZnYr8BOSl6F+zt3zbjKgRc21\nPP/yIbbtOcqMaeVhxxERGVcZjRm4+63DFm1KWfcEsCKDNrj7FWn2/wPgB5lkCct5Z9cBsPXVdq5Y\nNur4tojIpKM7kDPUPKuKqvJituxq1/0GIjLlqBhkqCgSYfHcOtq7ejlwZOxXM4mI5CMVgywsPSd5\nWenmnUdCTiIiMr5UDLKwZF5y3GDLrvaQk4iIjC8VgyzMmFbOzLpyXtrdzsBgPOw4IiLjRsUgS0vn\nTaenb5Cd+zvDjiIiMm5UDLK0ZF5y3EBdRSIylagYZGnx3FoiEdi8S4PIIjJ1qBhkqaKsmHNm1/DK\nvk66ewfCjiMiMi5UDMZgybzpxBMJfLcmrhORqUHFYAyWBpeY/nbn4ZCTiIiMDxWDMVgwZxrlpTFe\n3HFYU1OIyJSgYjAGsWgR58+fzqGOHloOHQ87jojIGVMxGKMLF8wAYNMOdRWJyOSnYjBGy+ZPJxKB\nTdsPhR1FROSMqRiMUXVFCQvmTGP7vg6OdfeHHUdE5IyoGJyBCxfUk0jAb15RV5GITG4qBmfg5LiB\nuopEZJJTMTgDcxoqqa8p5bevHNEspiIyqakYnIFIJMIF587gRO8AO/Z1hB1HRGTMVAzO0FBX0Qvq\nKhKRSUzF4AwtnltLWUmUjd6mu5FFZNJSMThDxbEoF547g0MdPew+eCzsOCIiY6JiMA4usQYANnhr\nyElERMYmlslGZnYncDmQAG5x9/Up69YAdwCDwFp3v320Nmb2N8A/AXXufixY1g88lfKUq9198AyP\nbcIsm19PSXER619q5Q/fPJ9IJBJ2JBGRrJy2GJjZSmChu68ws8XA3cCKlE3uAq4G9gG/NLMHgIaR\n2pjZ+4FZQMuwp+lw91VnfDQhKS2OcsGCGWx4qZW9bcdpnlkVdiQRkaxk0k20GvghgLtvBerMrAbA\nzOYDR9x9j7vHgbXB9unaPOjunyZ5tjClnOwqekldRSIy+WTSTdQIbEx53BYs6wx+tqWsawUWADNG\nauPu29I8R5mZ3QfMBR5w96+MFqiuroJYLJpB9JE1NFSPuW06b60p5+4fbeWFHYf58LUXjnk/ucg2\nHpQrO8qVvXzNVii5MhozGGa0DvF0607Xif4J4FskzxieMLMn3H1Duo3b20+cZnfpNTRU09bWNeb2\no1l6znSef/kQL2w9wJwZlVm3z2W2M6Fc2VGu7OVrtqmWa7QCkkk3UQvJM4AhTcD+NOvmBMtGa/M6\n7v5Vdz/m7seBx4DzM8iVdy45byYAG9VVJCKTTCbF4FHgOgAzWw60uHsXgLvvAmrMbJ6ZxYBrgu3T\nthnOku4zs0iwjyuBzWd2WOG46NwZxKJFPLv1oG5AE5FJ5bTdRO6+zsw2mtk6IA7cbGY3kLwC6EHg\nJuA7web3B+MC24a3ATCzTwNvI3nW8GMze9rdP2lme4Dngm0fcvfnxvcwJ0Z5aYyLFiavKtp1oItz\nZteEHUlEJCMZjRm4+63DFm1KWfcEp15qmq4N7v4F4AsjLP9UJjkmgyuWNrLhpVae3nxAxUBEJg3d\ngTzOls2fTlV5Mc9tOchgXNNai8jkoGIwzmLRIn5n8Uw6T/SzeWd72HFERDKiYpADK5YlL6R6evOB\nkJOIiGRGxSAH5s+uYVZdOc9va6O7dyDsOCIip6VikAORSIQVSxvpG4jz621tp28gIhIyFYMcuXzp\nLACe+k3ae+1ERPKGikGOzKyr4Lyza3lp91EOHBn79BkiIhNBxSCHVl40B4AnXhg+Y7eISH5RMcih\n5YsaqCov5le/2U//gO45EJH8pWKQQ8WxIq48v5Fj3f08/7IGkkUkf6kY5NhQV9Evnt8XchIRkfRU\nDHKscboGkkUk/6kYTAANJItIvlMxmADLFzVQXVHMky+20Ns3GHYcEZHXUTGYAMWxIt7yhjkc7xlg\n3W91E5qI5B8VgwnyluVnEYtGeHTDXuL6FjQRyTMqBhNkWmUJly2ZxcEjJ3hxx+Gw44iInELFYAK9\n7ZJmAH66fk/ISURETqViMIHOnlXN4rl1bH21nd0Hu8KOIyJykorBBLvqUp0diEj+UTGYYOcvqGd2\nfQXPbDnIoY7usOOIiAAqBhOuKBLhnSvmMhhP8ONndocdR0QEUDEIxWVLZjGztpwnX2yhvas37Dgi\nIsQy2cjM7gQuBxLALe6+PmXdGuAOYBBY6+63j9bGzP4G+Cegzt2PBcuuBz4KxIGvufvXx+fw8lO0\nqIh3rJjLN378Ej9+9lX+bM2isCOJSIE77ZmBma0EFrr7CuBG4K5hm9wFXAtcCVxlZkvStTGz9wOz\ngJaU/VcCnwHWAKuAj5nZ9DM8rrx3xbJG6mtK+eULLXQc7ws7jogUuEy6iVYDPwRw961AnZnVAJjZ\nfOCIu+9x9ziwNtg+XZsH3f3TJM8WhlwGrHf3DnfvBp4iWVimtFi0iHdcPpf+gTg/eU5jByISrkyK\nQSOQ+s0sbcGykda1ArPTtXH3kS6uT7ePKe+NFzRRV13K4xv3cvSYxg5EJDwZjRkMExnDutHaZL1t\nXV0FsVg0i12eqqGhesxtx9v1v3se//b9TTy6cR8Lz5mRV9lSKVd2lCt7+ZqtUHJlUgxaeO1MAKAJ\n2J9m3ZxgWd8obU63/znAM6MFam8f+5fENDRU09aWP3f/XnhOHY3TK3j0mVd595vnU5pN2Zwg+faa\nDVGu7ORrLsjfbFMt12gFJJNuokeB6wDMbDnQMtTd4+67gBozm2dmMeCaYPu0bUbwLHCpmdWaWRXJ\n8YInM8g1JUSLirh25QLiiQTfXLs17DgiUqBOe2bg7uvMbKOZrSN56efNZnYD0OHuDwI3Ad8JNr/f\n3bcB24a3ATCzTwNvI3km8GMze9rdP2lmtwI/ITmw/Dl37xjfw8xvyxfNYMGcGp7+zX7eelETC+ZM\nCzuSiBSYSGISzq3f1tY15tD5etq3bc9RvvTtX7PwrGncev1yIpH86S/K19dMubKTr7kgf7NNtVwN\nDdVpP1h0B3KeWNRcy2VLG3l5bwfPbj0YdhwRKTAqBnnkQ+9eRixaxPce305370DYcUSkgKgY5JHG\n+krecfnZHD3Wx8PrdoUdR0QKiIpBnnnH5XOZMa2Mn67fQ8uh42HHEZECoWKQZ0qKo/zJ6oUMxhN8\n+6fbmIwD/CIy+agY5KE3LJzB+fPr2fpqO7/6Tbp79URExo+KQR6KRCK8/2qjtCTKdx/bru88EJGc\nUzHIU/XTynjPW86lu3eAbz7ykrqLRCSnVAzy2MqLmjjv7Fo27TjMM1t074GI5I6KQR4rikS44R2L\nKSku4r6fblN3kYjkjIpBnptZW84fv+VcjvcM8LWHNhOPq7tIRMafisEksOoNc1i+qAHfc5QfPb0r\n7DgiMgWpGEwCkUiEG95+HtNrSvmfX+3i5b1Hw44kIlOMisEkUVVezId/bykJEnztoc10negLO5KI\nTCEqBpPIouZa3v3Gczjc2ctX/2czg/F42JFEZIpQMZhkrrliHm9YOIOtr7bz/Z/vCDuOiEwRKgaT\nTFEkwoeuWcLs+goeXb+HpzRdhYiMAxWDSai8NMZfX3sB5aUx/vsRZ/vegvqWUBHJARWDSapxegU3\nvXsp8XiCf/nBJvYf1nTXIjJ2KgaT2LL59Xzg7cbxngG+cv8mjh7THcoiMjYqBpPcmy5o4vffdA6H\nO3v45+9t4kSPvi5TRLKnYjAF/N4V81h1URO7W49x5/de0Pcni0jWVAymgEgkwnuvMlYsncWOlk7u\n/P4mevpUEEQkcyoGU0RRUYQb37mEy5bMYvveDv75+y+qIIhIxmKZbGRmdwKXAwngFndfn7JuDXAH\nMAisdffb07Uxs2bgXiAK7Afe5+69ZtYPPJXylKvdffCMj67AFBVF+NA1ixmMJ9jwUitf/s4LfOw9\nF1JVXhx2NBHJc6c9MzCzlcBCd18B3AjcNWyTu4BrgSuBq8xsyShtbgP+3d3fBGwH/jxY3uHuq1L+\nUyEYo2hREX/5riVcuayRnfs7+eK3NnKksyfsWCKS5zLpJloN/BDA3bcCdWZWA2Bm84Ej7r7H3ePA\n2mD7dG1WAQ8F+30YWDN+hyJDokVFfPCdi7nq0mb2Hz7BF7+1kX1tx8KOJSJ5LJNi0Ai0pTxuC5aN\ntK4VmD1Km0p37x22LUCZmd1nZk+Z2cezOwQZSVEkwh+/9VyuXTmfw529fOHejWzafijsWCKSpzIa\nMxgmMoZ1Iy1PXfYJ4FskxxeeMLMn3H1Duiepq6sgFoueNmg6DQ3VY26ba+Od7YZ3nc+C5un883d/\nzV0PvMgHr1nK769cQCQy2v/G3OcaL8qVnXzNBfmbrVByZVIMWnjtTACgieTg70jr5gTL+tK0OWZm\n5e7enbIt7v7VoQ3N7DHgfCBtMWhvP5FB7JE1NFTT1tY15va5lKts551Vw6euX85dD7zI3Q9vZvOO\nQ7z/aqO8NLO/BfL1NVOu7ORrLsjfbFMt12gFJJNuokeB6wDMbDnQ4u5dAO6+C6gxs3lmFgOuCbZP\n1+ZnJAebCX4+Ykn3mVkk2MeVwOasj1JGdc7sGj7zgUtZMKeGZ7cc5Lb/3sDeVo0jiEjSaYuBu68D\nNprZOpJXBd1sZjeY2R8Em9wEfAd4Erjf3beN1CbY9rPAB8zsSWA68N/u7sAe4DmSl5eudffnxu8Q\nZUhddSmf+rPlXP07zRw8coLbv7mBxzbuJZ5IhB1NREIWSUzCD4K2tq4xh87X0z6Y2GzPv9zG3T/a\nyvGeAZbMq+ODb19M/bSy0HNlQ7myk6+5IH+zTbVcDQ3VaQcLdQdygXrDwgZu/9BlXLCgni272vnM\n3c/yixf26SxBpECpGBSw2qpSbrnuAj749vNIJOCbjzhfvHcjuw/m319CIpJbKgYFLhKJ8KYLm/jC\nX1zOpefNZEdLJ7d9YwPfetTpPN4XdjwRmSAqBgIkB5dv+v1lfPw9FzKjtozHf72PT/3X0zz81E56\nNCW2yJSnYiCnWDa/ns9/6DKuf9siSmJFPPjkTv7ySz/jFy/sY2AwHnY8EcmRsdyBLFNcLFrE6ovP\n4opljfz42d38dMMevvmI8/BTu7j60mbefFETZSV664hMJTozkLTKS2P84Zvn81+3ruaqS5s50TPA\ndx/fzt/+xzoefOIVjSmITCH6805Oq35aOX+yeiHXXDGPx3+9l59t2MvD63bx42dfZfmiBlZdNAc7\nuzbr+Y5EJH+oGEjGqsqLedeV53D1pWfzq9/s5+fP7+O5ra08t7WVxukVrLyoicuXzGJaVWnYUUUk\nSyoGkrXSkiirLz6Lty6fw8t7O/jFC/vY8FIb9z++ne/9fDvnnV3HZUtmcbE1UFmmb1kTmQxUDGTM\nIpEIi5prWdRcy5+u7uOZLQd5bstBtr7aztZX27n3J87Sc6Zz0bkzuPDcGdRV64xBJF+pGMi4qK4o\n4W2XNPO2S5ppO9rNc1sP8uyWVl7ccZgXdxyGnzhnz6rionNnsPSc6Zwzu4ZYVNcviOQLFQMZdw21\n5bxzxTzeuWIerUe7eXH7ITZtP8RLu4+y++AxHnpqF6XFURaeNY3Fc+uws+s4e1aVioNIiFQMJKdm\n1paz5pJm1lzSTHfvAFt2tfPSq+1s3d3Ob3ce4bc7jwDJexvmNlYxf/Y05jfVsKCphvppZbpCSWSC\nqBjIhCkvjXGxNXCxNQDQcayXl3YfZdueo7zS0snOli527Os8uX11RTHNM6s4q6GKOQ2VNM+soqm+\nkpLisX/lqYiMTMVAQjOtqpTLlszisiWzAOjtH+TVA1280tLJKy0d7DrQxZZd7WzZ1X6yTSQCM+sq\naKwrZ+6caVSXxphVV87MunLqp5URLVJXk8hYqBhI3igtjp68OmlId+8A+w4dZ2/rMfa2HWNv23H2\ntR1j05ETbNpx+JT20aII9dPKaKgtp666lOnVpUyvKaOuuvTk4/LSmLqeREagYiB5rbw0xrlzpnHu\nnGmnLD/W3U8/EXznIVrbuzl4pJvWoydobe9mczAOMZLSkijTKkqoriimOvhZU1lCdXkx1ZXB8vIS\nqsqLKS+NUVYapUjFQwqAioFMSlXlxTQ0VFNX/vq3cG/fIEe6emjv6qW9q5cjwc/2zh6OdPXSeaKP\nwwd6GIyf/lvdIiQLUnlpjIqy4Oewx2UlUUpiRZQWRyktidLQdpye472UlESTy4qjlBRHKS0uoqRY\nxUXyk4qBTDmlJVFm11cyu74y7TaJRIITvQN0Hu+j60R/8F8fXSf66DzRz4mefk70DNDdO8CJ4L9D\nHd109w6ecb5YNEIsWhT8l/y9OFZ0cllxNEL0lGURiqNFyWXRIiJFyS6xoqJI8mckkvK4iKIIr61L\n2a52WgfHj/eeuj4SIRKJEIkkC9/J34f9HCpgRanLYcRtI5FIcgbMCKfuP2hD8FwEzwdQdqKP4z39\nBKuH1gbbnPr6DT2OEEndLGWf6dufvm3hFmoVAylIkUiEyrJiKsuKmV2febt4PEFP3wAnepIFort3\ngN7+QXr74/T2DdI3MEhxSYwjR7uD5YP0Bev7gt/7B+L0D8YZGEwwEPze293PQMoyfRN1fohEgAQp\nRWP0ApVmL+n3nVULiEaLuPGdi7m6oXq0JxwTFQORLBQVRagoK6ZilDmXGhqqaWsb+/dIJxIJBuOJ\nk8WhfyAe/J58HI8niCeSPwfjwc9hj4e2Obk+nqCispSOju6Tj4eeJxE8ZyKR8pOhx68tiyeSJSqe\nui2p25zaJh7sh8Tr2yQPlJO/l5RE6U35Rr3EsGqYCBYML5JD253c66k/Tm6QIP0+X7+v1xYUl8To\n6xsYtnzk5xxJulXDs2TWCqJFRdRUlozWeMxUDETyTCQSOdl9NJ7OtEjlUr5my9dcuaCLskVEJLMz\nAzO7E7ic5PnLLe6+PmXdGuAOYBBY6+63p2tjZs3AvUAU2A+8z917zex64KNAHPiau399vA5QRERO\n77RnBma2Eljo7iuAG4G7hm1yF3AtcCVwlZktGaXNbcC/u/ubgO3An5tZJfAZYA2wCviYmU0/4yMT\nEZGMZdJNtBr4IYC7bwXqzKyJR/V7AAAF50lEQVQGwMzmA0fcfY+7x4G1wfbp2qwCHgr2+zDJAnAZ\nsN7dO9y9G3iKZGEREZEJkkkxaATaUh63BctGWtcKzB6lTaW7955m26HlIiIyQcZyNdFoV9SmWzfS\n8my2PUVdXQWx2NhnrmzIwTW64yVfsylXdpQre/marVByZVIMWnjtTACgieTg70jr5gTL+tK0OWZm\n5UF30NC2I+3jmdECtbefyCD2yPL5UrF8zaZc2VGu7OVrtqmWa7QCkkk30aPAdQBmthxocfcuAHff\nBdSY2TwziwHXBNuna/MzkoPNBD8fAZ4FLjWzWjOrIjle8GSWxygiImcgMvwuvJGY2ZeAN5O89PNm\n4A1Ah7s/aGZvBv4+2PQBd//Hkdq4+yYzmw18EygDXgU+6O79ZnYd8LckL0P9V3f/9ngepIiIjC6j\nYiAiIlOb7kAWEREVAxERUTEQERFUDEREBBUDERFBxUBERCiwL7cZbSruELL8A/Amkv8Pvgi8C7gY\nOBxs8mV3/1EIuVYB3wc2B4t+A/wDI0w9PsG5bgTel7LoEmADUAkcD5b9L3ffOIGZlgH/A9zp7v+W\nL1O0p8l1D1AM9APvdfcDZtZPcmLIIavd/cy/5DnzXN9ghPd8Hrxe3wcagtXTSc6IcAfJfwtD7682\nd/+jHOca/hmxnhy+vwqmGKROq21mi4G7gRUhZXkLsCzIUg88DzwO/G93/39hZBrml+5+3dADM7uH\n5NTj3zezO4A/B/5zIgMFb/CvB3lWAu8BlpK8cfG3E5klyFAJ/CvwWMrioSnaT75OZvZNklO0/w7J\naVrWm9mD7n5kAnN9nuSHxPfM7Gbg48AnSd44uioXOTLMBcPe8ylT2of2eqV+yJvZ3cD/fW3VhL1e\nI31GPEYO31+F1E2UdiruEDwBDL3hjpL863bsM+/l3ipeP/V4mD4D3B5yhl7gHSTn1hqyivCnaB8p\n10eAB4Lf24D6HD5/OiPlGkk+vF4AmJkBte7+XA6fP52RPiNWkcP3V8GcGZCcDC+1C2FoWu3OiQ4S\nnIoPdW3cSPJ7IAaBvzKzj5Ocxvuv3P3QRGcLLDGzh0ieIn+OkaceD4WZXQrsCbo5AG4zsxnAVuCj\nwT+InHP3AWAgyDAk9CnaR8rl7scBzCxKcjqZ24JVZWZ2HzCX5FQyX5nIXIFT3vPkweuV4haSZw1D\nGs3sByQn3vz3XE6bk+Yz4upcvr8K6cxguNNOlZ1rZvZukv+j/4pkX+Ct7v5W4AXg70KK9TLJAvBu\n4AMku2ZS/2gI+3X7EPCN4Pd/Af7W3VPnzcoXY56iPReCQnAv8Li7D3WJfAL4MHAVcL2ZXTLBsTJ5\nz4f1epUAb3T3nweLDgP/B/hTkuN7twdzreU6R+pnRKpxf38V0pnBaFNxTzgzuxr4NPC77t7BqX2p\nDzHBffJD3H0fcH/wcIeZHSA5q+zwqcfDsgr4awB3fzBl+cPAH4cRKMW4TNGeI/cAL7v754YWuPtX\nh343s8eA80kOyk+IlKIEr73nf0B+vF4rgZPdQ8Gsy/cEDw+Z2QbgPHL4GTL8M8LMcvr+KqQzg7RT\ncU80M5sGfBm4Zmigx8weCL5GFJIfeBM+KBrkuN7MPhH83gjMIvmPYPjU42FkawKOuXufmUXM7Gdm\nVhusXkVIr1mKvJyiPbjapM/dP5uyzMzsvuB1jAW5NqfdSW5yjfSeD/31ClwKbBp6YGZvMbOvBL9X\nAhcB23L15CN9RpDj91dBzVo60rTaIeX4MMlT4tQ30z0kTwVPAMdIXiXTGkK2auA+oBYoIdll9Dwj\nTD0eQraLgc+7+9uDx+8BPkWyb3UfcKO7j/2bj7LP8k/APJKXa+4DrifZhRXaFO1pcs0EenhtfGyL\nu3/EzP4eeCvJfw8PufsXJjjXvwK3Muw9nwev1x+SfN//yt3vD7aLkbyqyEhe7PGf7n7PSPscp1wj\nfUZ8IMiQk/dXQRUDEREZWSF1E4mISBoqBiIiomIgIiIqBiIigoqBiIigYiAiIqgYiIgI8P8B0PnL\n/mqLXbgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history['learning_rate'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "X9uiFJ-ogBRi"
   },
   "source": [
    "### Loading a Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Doot7_0fY5tF"
   },
   "outputs": [],
   "source": [
    "unlabeled_images_test = pd.read_csv('gdrive/My Drive/dataML/test.csv')\n",
    "#unlabeled_images_test = pd.read_csv('test.csv')\n",
    "\n",
    "X_unlabeled = unlabeled_images_test.values.reshape(unlabeled_images_test.shape[0],28,28,1)/255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "dbtjpi8JgBRj",
    "outputId": "ee65fc89-ef7e-4b99-b2bb-4d24d4e518c3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from models_saving/my_model.ckpt\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    \n",
    "    # Restore the model\n",
    "    saver.restore(sess, 'models_saving/my_model.ckpt')\n",
    "    \n",
    "\n",
    "    # Fetch Back Results\n",
    "    label = sess.run(Y, feed_dict={X:X_unlabeled,drop_rate:0})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "MZ1xS7oJZYkV",
    "outputId": "6d483b80-a575-49e8-ce54-db9c91392100"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2, 0, 9, ..., 3, 9, 2])"
      ]
     },
     "execution_count": 74,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ojRKNc76gBRx"
   },
   "source": [
    "## Predict the unlabeled test sets using the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yQrGiou8gBRy"
   },
   "outputs": [],
   "source": [
    "imageId = np.arange(1,label.shape[0]+1).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9Oj02pY3gBR1"
   },
   "outputs": [],
   "source": [
    "prediction_pd = pd.DataFrame({'ImageId':imageId, 'Label':label})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VLjMgeXEgBR4"
   },
   "outputs": [],
   "source": [
    "prediction_pd.to_csv('gdrive/My Drive/dataML/out_cnn4.csv',sep = ',', index = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ivz_HMLnZ684"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "b6I1adl5gBQD",
    "xeMZR7ntgBQI",
    "LT78eGccgBQN",
    "4RIGVSojmy-R",
    "RX5PuSUmvRri",
    "X9uiFJ-ogBRi",
    "u13ffZdjrqQf"
   ],
   "name": "3 DL Multi-Layer CNN for DigitRecognizer.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
